Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
HOWTO: set up a local cddb/freedb server
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
supermihi
Guru
Guru


Joined: 09 Feb 2005
Posts: 348

PostPosted: Mon Mar 28, 2005 9:52 pm    Post subject: HOWTO: set up a local cddb/freedb server Reply with quote

HOWTO set up a local cddb/freedb server
to obtain automated Audio CD information without a permanent internet connection

Navigation:
0. Introduction
1. Pro's/Con's
2. Required Software
3. Installing the Database
4. Optional: setting up the webserver for http access
5. Installing the server software
6. Setting up xinetd
7. Client setup
8. Additional hints

0. Introduction

The CDDB protocol allows automated recognition of audio CDs by a special checksum. This checksum is sent to a server (e.g. freedb.org), that holds a huge database to find what album you are using. Then the server sends you information about track titles, album artist etc. This can be used for very comfortable automated cd ripping.
BUT:
Are you ripping a lot of your home music CD collection? Are you using a dialup internet connection being angry that you have to dial to your ISP a few seconds long for every disc to get the track names etc. filled in automatically? Or do you just think that there's nothing more cool around than your own, local cddb server providing information on hundreds of thousands of CDs? Well, then this is the right place for you - get ready to start!

*** WARNING! This setup is useful for local / home network use only!!! This document does NOT cover how to setup an official mirror of freedb. Security is ALMOST NOT covered here!! ***
1. Pro's/Con's:

There are many benefits from a local cddb/freedb server:
- you don't need an open internet connection for obtaining disc information from freedb
- you don't have to worry about security - nothing is transferred (unencrypted) over the internet, noone will ever know which kind of music you hear
- you can serve a local network with cddb infos
- you can tell everyone that you have your own cddb server running
- can be run with xinet.d - no performance issues
- there's a mailinglist available that tells you about database updates, so you don't have to worry about them

However, there are also some things you should keep in mind:
- you have to download the initial database, which is currently 315 megs (tar.rar) / 375 megs (tar.bz2), so you at least need a good friend who has a cheap connection
- you shoud download the updates that come about every month (don't need this if your collection contains only older discs)
- you will need some disc space - at least 2GB, depending on what filesystem you use
- unfortunately, there's currently no ebuild available for the server software

If you still think that you just WANT a local cddb server, continue!

2. Required Software

a) Software available in portage:
I recommend the use of xinetd which can start the cddb server only when it's needed:
Code:
root # emerge xinetd
root # /etc/init.d/xinetd start
root # rc-update add xinetd default

- to unpack the rar-packed database (which is much smaller in download size), you'll need rar or unrar.
Code:
root # emerge unrar

Filesystem considerations:
Since the database consits of hundreds of thousands of very small files, the filesystem you are using for storing the data has a great influence on how much disk space will actually be used (covering several hundreds of percents from reiser4 to vfat). I *highly* recommend to use reiser4, which uses disc space extremely efficiently with small files, or at least reiserfs. Reiser4 is not yet in the official kernel, you'll either need to patch your kernel or use one of the many kernel versions that include reiser4, like the mm-sources, nitro-sources etc. Also you will need the userspace utilities, which are part of portage:
Code:
emerge reiser4progs

**TIP** (thanks to Sanguinary) If you don't want to repartition your hdd, you could use the loop-mounting to create a filesystem on a single, large file in an existing filesystem. Then you can format this one with reiser4 or reiserfs (v3).
b) Software you have to download by hand:
- the cddb server software (currently v.1.5.1PL2) which is available at the freedb homepage: http://www.freedb.org/modules.php?name=Sections&sop=viewarticle&artid=7
- the cddb database which is also available at freedb.org: http://www.freedb.org/modules.php?name=Sections&sop=viewarticle&artid=12

3. Installing the database

First you should, like mentioned above, set up a partition that uses the reiser4 filesystem (if that's not possible for you, e.g. if you're using amd64, reiserfs (v3) will also do a good job, but far not as good as reiser4). What I did is to put my whole /var directory into a reiser4 partition, since there are a lot of small files.
Then unpack the database:
Code:
root # mkdir /var/db/cddb
root # rar p -inul freedb-complete-<latest version>.tar.rar | tar x -C /var/db/cddb/

if you took the bz2 version:
Code:
root # tar xjf freedb-complete-<latest version>.tar.bz2 -C /var/db/cddb

This will take a while! ;)

4. Optional: setting up the webserver for http access

The cddbd daemon can be also accessed via normal http using cgi-scripts. This is useful if you're having firewall issues or if you use a ripper that only supports this way (e.g. grip).
Therefore you need an apache webserver running:
Code:
root # emerge apache
root # /etc/init.d/apache2 start
root # rc-update add apache default

If you already have a webserver running on your machine, you will have to specify a cgi-path where cddbd can install its scripts. Otherwise, the default configuration will do fine (you might want to check the config file for security reasons).
5. Installing the server software

a). Extract the tarball
Code:
 root # tar -xvzf ccdbd-1.5.1PL2.tar.gz
root # cd cddbd-1.5.1PL2

b) Run the config script
Code:
root # sh config.sh

You can leave all of the questions to their defaults.
c) Compile the daemon.
Code:
root # make

d) Install using the install script
Code:
root # sh install.sh

The script will ask some questions that you should not leave as default:
    CD database directory: whatever you extracted the db to (in this howto /var/db/cddb)
    enable server access via HTTP? see section 4
    cddb cgi directory: This will show up if you answered the previous question with yes. In the default configuration of apache on gentoo, this should be set to /var/www/localhost/cgi-bin/
    enable logging of server accesses: this probably isn't neccesary for a local db, though it's good for testing reasons

All other options can be left as they are by default. The creation of the "fuzzy matching hash file" will also take a while, so grab a cup of coffee and think about your next steps.

6. Setting up xinetd

The "extended internet services daemon" xinetd lets you start services only when they are needed, thereby increasing both performance and security of your machine.
(NOTE: this is only for the direct cddbd access. If you also choosed HTTP acces to your server, you have to make sure that apache2 is running!)
Two steps are needed to get xinetd working with cddb:
1. Edit /etc/services
The file /etc/services sets up matches between ports and services.
Adding a line
Code:
cddbp           8880/tcp


lets xinetd know that any incoming tcp request on port 8880 should be served by cddbp.
Now we should tell xinetd what cddbp is. Therefore we create a file /etc/xinetd.d/cddbp with the following content:
Code:
service cddbp
{
        disable             = no
        socket_type         = stream
        protocol            = tcp
        wait                = no
        user                = root
        server              = /usr/local/bin/cddbd # set this to wherever you installed the binary
}

Now reload the xinetd configuration and test if all goes well:
Code:
root # /etc/init.d/xinetd reload
root # telnet localhost 8880

you should get some response now - congratulations! You now have your own cddb server.

7. Client setup

To set up your clients using native cddb protocol on port 8880 (not http access), simply fill in the hostname or IP your local cddbd server is running at (localhost if it's the same machine the client runs on).

If you act as an internet gateway, it should also be possible to redirect outgoing CDDB requests from the local net to your server with iptables.

If you're using HTTP access, server is same as above and cgi-path is "cgi-bin/cddb.cgi".

8. Additional hints

a) If you set up HTTP access and used the default apache config, you'll need to change the permissions of the cddb.cgi script:
Code:
root # chmod a+rx /var/www/cgi-bin/cddb.cgi
root # chmod u+s /var/www/cgi-bin/cddb.cgi

You can try if it works by accessing the following URL in your browser: http://localhost/cgi-bin/cddb.cgi?cmd=stat&hello=me+host+none+1.0&proto=1

b) Be sure to register to the fdb-announce@freedb.org mailinglist, so you get mails everytime an update of the database gets released.

c) Further information can be found in the CDDBD_HOWTO file in your extracted server software package.

That's it! Please post comments, suggestions etc!!
_________________
"You may say I'm a dreamer, but I'm not the only one."


Last edited by supermihi on Sun Jan 08, 2006 6:20 pm; edited 1 time in total
Back to top
View user's profile Send private message
beandog
Developer
Developer


Joined: 04 May 2003
Posts: 2007
Location: /usa/utah

PostPosted: Sun May 15, 2005 6:29 pm    Post subject: Reply with quote

Thanks for the great guide -- this is a good condensed alternative to the very lengthy, but well documented, HOWTO. :)

I'm just having one issue with my server. I'm getting errors when I access the cddb.cgi. Here's what my apache error_log is spitting out:

Code:
[error] [client 127.0.0.1] Premature end of script headers: cddb.cgi


I've tried your user permissions on the cgi, I've tried setting it to 777, and tried it in both /var/www/localhost/cgi-bin, /home/public_html/cddb and /home/cddb/public_html/cgi-bin and nothing works.

Other than that, it works fine if I access it over port 8880. Any ideas?
_________________
If it ain't broke, tweak it. packages | dvds | blurays | blog | home
Back to top
View user's profile Send private message
supermihi
Guru
Guru


Joined: 09 Feb 2005
Posts: 348

PostPosted: Mon May 16, 2005 6:46 am    Post subject: Reply with quote

not really, since I don't know anything about cgi / apache. Sorry!
_________________
"You may say I'm a dreamer, but I'm not the only one."
Back to top
View user's profile Send private message
Sanguinary
n00b
n00b


Joined: 17 Oct 2004
Posts: 16

PostPosted: Sun Jan 08, 2006 6:11 pm    Post subject: Re: HOWTO: set up a local cddb/freedb server Reply with quote

supermihi wrote:

Filesystem considerations:
Since the database consits of hundreds of thousands of very small files, the filesystem you are using for storing the data has a great influence on how much disk space will actually be used (covering several hundreds of percents from reiser4 to vfat). I *highly* recommend to use reiser4, which uses disc space extremely efficiently with small files, or at least reiserfs. Reiser4 is not yet in the official kernel, you'll either need to patch your kernel or use one of the many kernel versions that include reiser4, like the mm-sources, nitro-sources etc.


Resurrecting an old thread here, but I used this to put a local freedb server on an old Cobalt Qube 2, and my experience taught me two things:

1. As a practical matter, you can't put the freedb database on an ext3 filesystem. The December 2005 version of the database is something like 1.5 million little text files. I gave up trying to untar the thing onto an ext3 filesystem when it took more than 36 hours without getting halfway through (and this wasn't on the Qube, so it wasn't a matter of the miniscule processing power that system has).

2. If you don't have reiserfs on your system and you don't want to repartition, loopback is your friend. Just as an illustration, I created and mounted a 5 gig file with a reiser3 filesystem, and untarring the same database took about 25 minutes.
Back to top
View user's profile Send private message
supermihi
Guru
Guru


Joined: 09 Feb 2005
Posts: 348

PostPosted: Sun Jan 08, 2006 6:18 pm    Post subject: Reply with quote

Good tip! Thanks, I'll mention that in the tutorial.
_________________
"You may say I'm a dreamer, but I'm not the only one."
Back to top
View user's profile Send private message
Gruffi
Apprentice
Apprentice


Joined: 15 Aug 2003
Posts: 209
Location: Antwerpen - Flanders - Belgium

PostPosted: Sun Nov 15, 2009 4:19 pm    Post subject: Reply with quote

You can also use squashfs. (emerge squashfs-tools)
The whole database was 11GB on ext3 but compressed with squashfs it was 950MB
Code:
cd /mnt/storage/cddb
mksquashfs -always-use-fragments -no-duplicates . ../cddb.squashed
rm -r /mnt/storage/cddb/* #deleting the files takes forever
mount -o loop -t squashfs /mnt/storage/cddb.squashed /mnt/storage/cddb


works just fine, mksquashfs took 80 minutes tho :mrgreen:
And thanks for the how-to! :D
_________________
... and we will show Microsoft, that they cannot take whatever they want. And that Free Software is our software!
Back to top
View user's profile Send private message
jlpoole
Apprentice
Apprentice


Joined: 01 Nov 2005
Posts: 226
Location: Napa, CA

PostPosted: Wed Sep 21, 2011 2:59 am    Post subject: Reply with quote

Here's a gotcha if you're not famliar with xinet: it appears xinet does not tolerate comments that start after column #1. For example, the above FAQ recommended:
Code:
server              = /usr/local/bin/cddbd # set this to wherever you installed the binary

I left the comment in (having just copy/pasted the recommend file contents) as is since /usr/local/bin/cddbd was indeed the server executable. Leaving the comment "# set this..." in the file apparently causes xinetd to quietly reject the entry. When I attempted to telnet to the server, as recommended, the connection failed:
Code:
themis cddbd-1.5.2 # telnet localhost:8880
telnet: could not resolve localhost:8880/telnet: Name or service not known
themis cddbd-1.5.2 #


I then modified /etc/xinet.d/cddbp by removing the comment text as follows:
Code:
themis cddbd-1.5.2 # cat /etc/xinetd.d/cddbp
service cddbp
{
        disable             = no
        socket_type         = stream
        protocol            = tcp
        wait                = no
        user                = apache
        server              = /usr/local/bin/cddbd
}
themis cddbd-1.5.2 #


Note: I also replaced root with apache since I wanted the cgi script to work. Using the modified file, telnet successfully worked.
Code:
themis cddbd-1.5.2 # telnet localhost 8880
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
201 themis CDDBP server v1.5.2PL0 ready at Tue Sep 20 19:39:56 2011

exit
Connection closed by foreign host.
themis cddbd-1.5.2 #


Lastly, don't even think about unzipping the database (which is now about 800 MBs) thinking you'll copy the unzipped files to their final destination -- I have a pretty powerful laptop and after several hours, canceled the "cp -r ..." and just moved the tar.bz2 to the directory where I wanted the database to reside and unzipped it there. I think I may investigate the suggestion of using the reiserfs on a file (rather than create another partition), but that's for another adventure.

Also, if you are on ext3 and have the database staged, remember when you "updatedb" to exclude the directory, as in:
Code:
updatedb -e /usr/local/cddb


Finally, just for statistics sake, here's the output from my apache session (which worked right after the install):
Code:
   210 OK, status information follows (until terminating `.')
   Server status:
      current proto: 1
      max proto: 6
      interface: http
      gets: no
      puts: no
      updates: no
      posting: no
      validation: accepted
      quotes: no
      strip ext: no
      secure: yes
      current users: 1
      max users: 100
   Database entries: 3088945
   Database entries by category:
      newage: 121127
      rock: 799682
      folk: 250973
      data: 70241
      jazz: 183896
      misc: 967927
      classical: 279550
      blues: 159920
      reggae: 42189
      country: 88599
      soundtrack: 124841
   .


Yeah! Classical music is alive and well represented, e.g. near 280,000 entries.
Back to top
View user's profile Send private message
jlpoole
Apprentice
Apprentice


Joined: 01 Nov 2005
Posts: 226
Location: Napa, CA

PostPosted: Wed Sep 21, 2011 3:04 am    Post subject: Reply with quote

Also, people may wonder why I went to the trouble to set up a local server. Basically, I wanted to be able to create MP3s and wave format files and found abcde to best fit the bill for my needs. Unfortunately, when it tries to connect to the cddb server that everyone else connects to, if it doesn't get the information sought, it just proceeds using generic naming of the files. The failure to get the appropriate naming due to a server being oversubscribed caused me to undertake the trouble of having my own local server so my files would always be suitably named. When I checked the statistics of the main free server, there were 100 connections -- the limit, so I was competing for a limited resource.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum