Sunday, September 9, 2012

Howto : XBMC share and sync your Media library between multiple XBMC instances using Samba/NFS and Mysql




Last update:
09/19/2012: FAQ - deleting databases if required 


If you have multiple XBMC installations (or if you're planing to), you should be interested in sharing and syncing your Media Library between your installations.

When you have multiple XBMC running in your Network, and one central storage, this will really really simplify your life ^^

Advantages:

  • Share watched and unwatched status for your media on all devices
  • Stop watching a movie or TV show in one room then finish watching it in another room automatically
  • Only one library to maintain for all devices
  • Consolidate thumbnails and fanart to save hard drive space (only with XBMC Frodo v12) 

Main source:
http://wiki.xbmc.org/index.php?title=HOW-TO:Sync_multiple_libraries/Share_your_files

You'll find in this Wiki page all required information to achieve this, but i wanted to share my own experience to help you implementing this great feature!

I recommend you to use XBMC Frodo v12, even if not yet stable this version will work better in this configuration as it is able to deal itself with shared Library data like thumbnails and fanart.

With XBMC Eden (v11) sharing the Library works too but you have to synchronize yourself Thumbnails Datas.

Let's start ^^


1. Share your files



First consider your sharing protocol:

You may use 2 major file sharing protocols to share your Media Library, SMB protocol which is the most implemented (used by Windows, implemented in Linux through Samba) or NFS. (Network File System, more known in Unix World)

Choosing is simple, if you have or plan to have Windows XBMC clients, use SMB with Samba.
If you only have Linux Based XBMC, you should use NFS because of betters speed performances.

Note: NFS will also be supported in XBMC under Windows, but you could lack performances



To share your file with SMB using Samba:


If it's not already installed, you have to get samba up and running to share your Multimedia files between your XBMC instances:

On Ubuntu based systems:
$ sudo apt-get install samba


Share your files:

Setup simple shares for your media library, edit "/etc/samba/smb.conf" and add your share, example:
[MY_SHARE]
path = /media/MY_VOLUME
available = yes
browsable = yes
public = yes
writable = yes


Restart samba:
$ sudo service samba restart



To share your file with SMB using NFS:


On Ubuntu based systems, install NFS Server components:
$ sudo apt-get install nfs-kernel-server


Edit "/etc/exports" and configure your shared directory, example (adapt MY_VOLUME  and your Local Network IP):
/media/MY_VOLUME xxx.xxx.xxx.xxx/255.255.0.0(rw,all_squash,insecure,no_subtree_check,async)


Note: "async" option is not recommended for secured write operations, but an XBMC client should not expected to have write operations), this option may enhance NFS performances)
You can also choose "sync".

Restart NFS Server:

sudo /etc/init.d/nfs-kernel-server restart


2. Install and setup Mysql for your Database


Extracted from main XBMC Wiki page, follow these instructions:

Install Mysql:
$ sudo apt-get install mysql


- Create a main admin password when asked

- After installation has complete, edit "/etc/mysql/my.cnf" and do changes as follows:

Look for the line "bind-address" and change "127.0.01" with your host IP address.
You can also simply comment this line to allow access from anywhere.


Restart Mysql:
$ sudo service mysql restart



Login to Mysql using client (adapt password):
$ mysql -u root -p


Create the XBMC user,  set password and rights, enter commands as follows:
CREATE USER 'xbmc' IDENTIFIED BY 'xbmc';
GRANT ALL ON *.* TO 'xbmc';


Exit mysql client. (exit)


3. Set up XBMC


At this point, if you already have set your Media Library inside your main XBMC installation, you have 2 possibilities:

  • Converting your existing Media Library
  • Re-creating your Media Library from scratch

To be honest, even if you have a big Media Library, converting it will be enough complicated for not so much benefits.

This is my opinion, but i would recommend to completely re-create your Library from scratch, this is much easier than converting it, moreover that you won't loose your existing thumbnails and fanart.

If you want to convert it, follow instructions at:
http://wiki.xbmc.org/index.php?title=HOW-TO:Sync_multiple_libraries/Setting_up_XBMC

If you want to re-create it from scratch (my advice), do:

- In XBMC, Go to Video and delete each source you have previously set, say Yes when XBMC asks if you want to purge these media from your Library

- Go to Parameters > Video and Clean your Library

- Configure your mysql settings by editing your advancedsettings.xml (~/.xbmc/userdata/advancedsettings.xml) and adding:

Adapt "xxx.xxx.xxx.xxx" with your main IP Address:
    <videodatabase>
        <type>mysql</type>
        <host>xxx.xxx.xxx.xxx</host>
        <port>3306</port>
        <user>xbmc</user>
        <pass>xbmc</pass>
    </videodatabase> 

    <musicdatabase>
        <type>mysql</type>
        <host>xxx.xxx.xxx.xxx</host>
        <port>3306</port>
        <user>xbmc</user>
        <pass>xbmc</pass>
    </musicdatabase>


- Restart XBMC

- Go in Video and re-set your Media Sources using Samba/NFS shares instead of Local paths

This is very important, right now any source you want to add in your Media Library must be done using Samba shares, files paths must use UNC patchs to be able to be shared.

Here's some screenshots to explain:

When you will add your source, ensure your choose a SMB or NFS Share:




And navigate until your Media Share, and finally set the scrapper as always.

Any media found will automatically be added to the Mysql Database.


4- Setup other XBMC instances (clients)


- Stop XBMC

First thing to do with any XBMC client instance: Setup advancedsettings.xml

- Edit each of your client advancedsettings.xml and ensure your add the mysql section (adapt your IP Adress), this is exactly the same code than for the main instance:


    <videodatabase>
        <type>mysql</type>
        <host>xxx.xxx.xxx.xxx</host>
        <port>3306</port>
        <user>xbmc</user>
        <pass>xbmc</pass>
    </videodatabase> 

    <musicdatabase>
        <type>mysql</type>
        <host>xxx.xxx.xxx.xxx</host>
        <port>3306</port>
        <user>xbmc</user>
        <pass>xbmc</pass>
    </musicdatabase>


- Restart XBMC (on clients)


Then setup your sources on clients:

For any other XBMC instances, just copy your main XBMC instance sources.xm file to your clients.

This is very important as all of your XBMC instances have to use the same source settings, using SMB shares.

To achieve this, nothing more simple :-)

"sources.xml" is located in "~/.xbmc/userdata/sources.xml"

If you have playlists and favorites, you can also just copy them from your main XBMC instance to client instances:


  • Playlists are located in "~/.xbmc/userdata/playlists"
  • favourites.xml in "~/.xbmc/userdata/favourites.xml"


Final before launching your XBMC client:

About fanarts and other datas:

- If your clients are running XBMC Frodo (v12), then you don't have nothing to do.
XBMC will take care itself with fanart and other data.
Any future added Media will also be available in clients, including Thumbnails and fanart.

- If you have clients with XBMC Eden (v11), you have to copy the Thumbnails directory from your main XBMC instance: ~/.xbmc/userdata/Thumbnails/

For future media add, it will be available on both main XBMC and clients, but you won't get Thumbnails and fanart in clients.

For this to work, you will have to set scrappers in clients too (as in main XBMC instance) and scan for new medias each time you added some. (or enable scan at launch time)





And finally start XBMC.

Magical you'll get your full Media Library centralized and always synced :-)

Off course you main XBMC instance acts as the main server, clients won't be able to access to Media Library if the server is down...



FAQ



- How to re-initialize my database in case of trouble:

In case of trouble you may have to totally re-reinitialize your database. (if you're running thousands of hard tests ^^^).

To do so, simply delete all XBMC databases and re-create from scratch:^

Note: don't worry you won't loose nothing


Login to Mysql using client (adapt password):
$ mysql -u root -p



List XBMC Databases:
mysql > show databases;



Then for each database, delete it, example:
mysql > drop database xbmc_video;


And restart from importing your Media ^^







12 comments:

  1. I have a question
    Does the mysql database should appear in two databases (video, audio)?
    Or maybe each instance of XBMC should create your database?
    For me it is like this:
    Three instances of XBMC, and the effect of:
    xbmc_music18
    xbmc_music20
    xbmc_music27
    xbmc_video60
    xbmc_video61
    xbmc_video67

    ReplyDelete
    Replies
    1. Your question is a good question.
      I can observe i have following tables:

      MyMusic18
      MyMusic27
      MyVideos60
      MyVideos67
      MyVideos78

      As i've tested different clients, i guess that it is possible XBMC clients automatically created these database, i have to check and test that.
      But it's certain that XBMC creates itself its own Database as i did not. (at least the server)

      Another approach is to manually create Database when initially configuring SQL:
      CREATE database xbmc_video;
      CREATE database xbmc_music;
      and granting rights (GRANT ALL ON *.* TO 'xbmc';)


      Then in advancedsettings.xml for both server and client, you specify your database name for each section (video and music)using :

      note: i can't put xml separator in comment don't forget it

      music section:
      name xbmc_music name

      video section:
      name xbmc_video name

      This approach could seem more "clean" and than letting XBMC to decide itself.
      Therefore, the first method without specifying Databases and letting XBMC dealing with it comes from the Official XBMC Wiki.

      I will test that and revert.

      Delete
    2. After some tests on various clients, i can confirm that the method presented in my post is fully functional and recommended.

      Anyone wanting to synchronize its XBMC Library should let XBMC to decide how to create it's own Databases, new connected clients won't create some and will connect with no issues to existing Databases

      Delete
  2. Nice write up. I have a question. If my main XMBC is running windows, media is stored in a NAS which supports both NFS and SMB and using Rpi as media player connected to my TV. Do you recommend that I use NFS or SMB as my media sources when adding sources? I was thinking NFS since it is suppose to have better performance.

    ReplyDelete
    Replies
    1. Thanks, no easy to say as it may depends on many factors (like the quality of your NAS NFS implementation and so on)
      As you are running Windows as the main XBMC, i think i would recommend SMB.

      But if you're courageous you can give a try to both ^^ Get a big big HD file (like a native blueray of 50 GB) to use a your main playback test, test playing back with NFS and SMB from your main XBMC under Windows and compare.

      If you don't have any issue nor lack of performances in NFS, then keep NFS.

      Note: the test i mentioned is for your main XBMC under Windows, don't think about playing back a 50 GB movie in Raspi as for now

      Delete
  3. Hi, interesting write up - do I really need a "main" XBMC machine... in my setup I have a Linux (Ubuntu) file server with mysql and nfs installed but not XBMC. I then want to connect two clients - presumably in this setup both XBMC can work independently and the "main" XBMC is in fact the NAS?

    ReplyDelete
    Replies
    1. Hi, thanks.

      Well it's up to you, you can off course just simply use you main computer as network file server.

      The advantage of sharing using Mysql is that you won't have to deal with updating media library on each XBMC instance, with time and library growing you often have some media not being recognized by scrappers and you have to check it manually.

      If you share your media library, you do this job only one time :-)

      Disadvantage is that you may have to deal with some issues if you don't strictly use the exact same XBMC version.

      In my case i have:

      - One main XBMC running on a Zotac HTPC with XBMC XVBA (XBMC 12.X FRODO)
      - One Raspberry running OpenELEC (XBMC 12.X FRODO)

      There are same major XBMC releases but they don't use the mysql db version (xbmc internal mechanism), the first is using version MyVideos68 and the second version MyVideos73 (mysql database names)

      The second version mysql (so the second mysql db) is created automatically when OpenELEC starts (if the db does not exist) and will be populated by data of previous db by XBMC.
      In this case, i have same media available on both.

      Later i add media and update my library using the main XBMC instance, but as it's using a previous version in comparison with OpenELEC, my Raspberry doesn't know anything about this. (it's using the newer version)

      To solve this, i have to delete the OpenELEC mysql database version and restart OpenELEC, then the database is re-created and re-populated with up to date data.

      This is a temporarily issue i have because my XBMC versions are not exactly using the same db version.
      Previously i did not have this problem because they were using same db version.

      If you have different hardware and can't use the same XBMC flavor, you could get in this issue.

      So it's up to you, deleting from time to time the newer mysql db is not an hard thing but you have to know what you do, and you have to think about it.

      If you have same king of hardware and you can use the same XBMC source, then you definitively should choose to share your db using Mysql, this is very great not to have to deal with updating and checking media library on mode than one instance!

      Regards,

      Delete
    2. Hi, good job, very interesting guide indeed.
      Have you tried not using mysql and just putting the .xbmc/userdata/ directory on the NFS server and then have all the xbmc instances mount that share? Do you thing that this would solve the incompatibility issues between the dbs of different xbmc versions or the sqlite db also has the versioning issues you mentioned?

      Another way of doing this would be every client to use its own userdata directory and rsync it with the main xbmc client.

      Another solution i could thing of would be to use mysql dbs locally on each node and then use mysql replication between the various clients and the main xbmc box. All clients would have the exact same dbs as the main box. For this you would need to have the same mysql-server version running on all the nodes and all the nodes but the main one would have to be configured not to be able to alter the db. (This would not be true if there were only to nodes and the replication was implemented in both directions.) But again this would not solve the different xbmc db versions problem you mentioned.

      Just some thoughts ...

      Delete
    3. Hi, Many thanks for your comment ^^

      Your comment is very interesting, to answer to some of your questions / ideas:

      - First and important thing, As far as you take care about keeping your various system up to date, XBMC native sharing with Mysql works very fine and i'm very satisfied with this now :-)

      - No i have not tried myself sharing and mounting the .xbmc/userdata dir using NFS, it should in deed work but may cause some issue or complication if you need specific settings for some of your devices.
      Also, concurrent write attempts could cause unexpected issue.

      But i think this should almost work

      - Using rsync is in deed a simple solution that will work for sure, just some home work to do

      - Mysql replication could as you say be a solution too, but enough complex in my opinion for a home installation ^^

      So finally, in my opinion the better and fully functional solution is definitively using Mysql with XBMC library share :-)

      Delete
  4. This comment has been removed by the author.

    ReplyDelete
  5. Nice guide, but what I did for the Thumbnails & Artwork is to add the following to the advancedsettings.xml in order to have them only scraped once.


    < pathsubstitution>
    < substitute>
    < from>special://masterprofile/Thumbnails/< /from>
    < to>smb://xxx.xxx.xxx.xxx/XBMC/Thumbnails< /to>
    < /substitute>
    < /pathsubstitution>

    ReplyDelete

Please feel free to comment ^^