| View previous topic :: View next topic |
| Author |
Message |
supermihi
Guru
Joined: 09 Feb 2005
Posts: 348
|
 Posted: Mon Mar 28, 2005 9:52 pm Post subject: HOWTO: set up a local cddb/freedb server |
 |
|
HOWTO set up a local cddb/freedb server
to obtain automated Audio CD information without a permanent internet connection
Navigation:
0. Introduction
1. Pro's/Con's
2. Required Software
3. Installing the Database
4. Optional: setting up the webserver for http access
5. Installing the server software
6. Setting up xinetd
7. Client setup
8. Additional hints
0. Introduction
The CDDB protocol allows automated recognition of audio CDs by a special checksum. This checksum is sent to a server (e.g. freedb.org), that holds a huge database to find what album you are using. Then the server sends you information about track titles, album artist etc. This can be used for very comfortable automated cd ripping.
BUT:
Are you ripping a lot of your home music CD collection? Are you using a dialup internet connection being angry that you have to dial to your ISP a few seconds long for every disc to get the track names etc. filled in automatically? Or do you just think that there's nothing more cool around than your own, local cddb server providing information on hundreds of thousands of CDs? Well, then this is the right place for you - get ready to start!
*** WARNING! This setup is useful for local / home network use only!!! This document does NOT cover how to setup an official mirror of freedb. Security is ALMOST NOT covered here!! ***
1. Pro's/Con's:
There are many benefits from a local cddb/freedb server:
- you don't need an open internet connection for obtaining disc information from freedb
- you don't have to worry about security - nothing is transferred (unencrypted) over the internet, noone will ever know which kind of music you hear
- you can serve a local network with cddb infos
- you can tell everyone that you have your own cddb server running
- can be run with xinet.d - no performance issues
- there's a mailinglist available that tells you about database updates, so you don't have to worry about them
However, there are also some things you should keep in mind:
- you have to download the initial database, which is currently 315 megs (tar.rar) / 375 megs (tar.bz2), so you at least need a good friend who has a cheap connection
- you shoud download the updates that come about every month (don't need this if your collection contains only older discs)
- you will need some disc space - at least 2GB, depending on what filesystem you use
- unfortunately, there's currently no ebuild available for the server software
If you still think that you just WANT a local cddb server, continue!
2. Required Software
a) Software available in portage:
I recommend the use of xinetd which can start the cddb server only when it's needed:
| Code: | root # emerge xinetd
root # /etc/init.d/xinetd start
root # rc-update add xinetd default |
- to unpack the rar-packed database (which is much smaller in download size), you'll need rar or unrar.
| Code: | | root # emerge unrar |
Filesystem considerations:
Since the database consits of hundreds of thousands of very small files, the filesystem you are using for storing the data has a great influence on how much disk space will actually be used (covering several hundreds of percents from reiser4 to vfat). I *highly* recommend to use reiser4, which uses disc space extremely efficiently with small files, or at least reiserfs. Reiser4 is not yet in the official kernel, you'll either need to patch your kernel or use one of the many kernel versions that include reiser4, like the mm-sources, nitro-sources etc. Also you will need the userspace utilities, which are part of portage:
| Code: | | emerge reiser4progs |
**TIP** (thanks to Sanguinary) If you don't want to repartition your hdd, you could use the loop-mounting to create a filesystem on a single, large file in an existing filesystem. Then you can format this one with reiser4 or reiserfs (v3).
b) Software you have to download by hand:
- the cddb server software (currently v.1.5.1PL2) which is available at the freedb homepage: http://www.freedb.org/modules.php?name=Sections&sop=viewarticle&artid=7
- the cddb database which is also available at freedb.org: http://www.freedb.org/modules.php?name=Sections&sop=viewarticle&artid=12
3. Installing the database
First you should, like mentioned above, set up a partition that uses the reiser4 filesystem (if that's not possible for you, e.g. if you're using amd64, reiserfs (v3) will also do a good job, but far not as good as reiser4). What I did is to put my whole /var directory into a reiser4 partition, since there are a lot of small files.
Then unpack the database:
| Code: | root # mkdir /var/db/cddb
root # rar p -inul freedb-complete-<latest version>.tar.rar | tar x -C /var/db/cddb/
|
if you took the bz2 version:
| Code: | | root # tar xjf freedb-complete-<latest version>.tar.bz2 -C /var/db/cddb |
This will take a while! 
4. Optional: setting up the webserver for http access
The cddbd daemon can be also accessed via normal http using cgi-scripts. This is useful if you're having firewall issues or if you use a ripper that only supports this way (e.g. grip).
Therefore you need an apache webserver running:
| Code: | root # emerge apache
root # /etc/init.d/apache2 start
root # rc-update add apache default |
If you already have a webserver running on your machine, you will have to specify a cgi-path where cddbd can install its scripts. Otherwise, the default configuration will do fine (you might want to check the config file for security reasons).
5. Installing the server software
a). Extract the tarball
| Code: | root # tar -xvzf ccdbd-1.5.1PL2.tar.gz
root # cd cddbd-1.5.1PL2 |
b) Run the config script
| Code: | | root # sh config.sh |
You can leave all of the questions to their defaults.
c) Compile the daemon.
d) Install using the install script
| Code: | | root # sh install.sh |
The script will ask some questions that you should not leave as default:
CD database directory: whatever you extracted the db to (in this howto /var/db/cddb)
enable server access via HTTP? see section 4
cddb cgi directory: This will show up if you answered the previous question with yes. In the default configuration of apache on gentoo, this should be set to /var/www/localhost/cgi-bin/
enable logging of server accesses: this probably isn't neccesary for a local db, though it's good for testing reasons
All other options can be left as they are by default. The creation of the "fuzzy matching hash file" will also take a while, so grab a cup of coffee and think about your next steps.
6. Setting up xinetd
The "extended internet services daemon" xinetd lets you start services only when they are needed, thereby increasing both performance and security of your machine.
(NOTE: this is only for the direct cddbd access. If you also choosed HTTP acces to your server, you have to make sure that apache2 is running!)
Two steps are needed to get xinetd working with cddb:
1. Edit /etc/services
The file /etc/services sets up matches between ports and services.
Adding a line
lets xinetd know that any incoming tcp request on port 8880 should be served by cddbp.
Now we should tell xinetd what cddbp is. Therefore we create a file /etc/xinetd.d/cddbp with the following content:
| Code: | service cddbp
{
disable = no
socket_type = stream
protocol = tcp
wait = no
user = root
server = /usr/local/bin/cddbd # set this to wherever you installed the binary
} |
Now reload the xinetd configuration and test if all goes well:
| Code: | root # /etc/init.d/xinetd reload
root # telnet localhost 8880
|
you should get some response now - congratulations! You now have your own cddb server.
7. Client setup
To set up your clients using native cddb protocol on port 8880 (not http access), simply fill in the hostname or IP your local cddbd server is running at (localhost if it's the same machine the client runs on).
If you act as an internet gateway, it should also be possible to redirect outgoing CDDB requests from the local net to your server with iptables.
If you're using HTTP access, server is same as above and cgi-path is "cgi-bin/cddb.cgi".
8. Additional hints
a) If you set up HTTP access and used the default apache config, you'll need to change the permissions of the cddb.cgi script:
| Code: | root # chmod a+rx /var/www/cgi-bin/cddb.cgi
root # chmod u+s /var/www/cgi-bin/cddb.cgi
|
You can try if it works by accessing the following URL in your browser: http://localhost/cgi-bin/cddb.cgi?cmd=stat&hello=me+host+none+1.0&proto=1
b) Be sure to register to the fdb-announce@freedb.org mailinglist, so you get mails everytime an update of the database gets released.
c) Further information can be found in the CDDBD_HOWTO file in your extracted server software package.
That's it! Please post comments, suggestions etc!!
_________________
"You may say I'm a dreamer, but I'm not the only one."
Last edited by supermihi on Sun Jan 08, 2006 6:20 pm; edited 1 time in total |
|
| Back to top |
|
 |
beandog
Developer
Joined: 04 May 2003
Posts: 1948
Location: /usa/utah
|
| Posted: Sun May 15, 2005 6:29 pm Post subject: |
|
|
Thanks for the great guide -- this is a good condensed alternative to the very lengthy, but well documented, HOWTO. 
I'm just having one issue with my server. I'm getting errors when I access the cddb.cgi. Here's what my apache error_log is spitting out:
| Code: | | [error] [client 127.0.0.1] Premature end of script headers: cddb.cgi |
I've tried your user permissions on the cgi, I've tried setting it to 777, and tried it in both /var/www/localhost/cgi-bin, /home/public_html/cddb and /home/cddb/public_html/cgi-bin and nothing works.
Other than that, it works fine if I access it over port 8880. Any ideas?
_________________
If it ain't broke, tweak it. packages | planet | blog | home |
|
| Back to top |
|
|
supermihi
Guru
Joined: 09 Feb 2005
Posts: 348
|
| Posted: Mon May 16, 2005 6:46 am Post subject: |
|
|
not really, since I don't know anything about cgi / apache. Sorry!
_________________
"You may say I'm a dreamer, but I'm not the only one." |
|
| Back to top |
|
|
Sanguinary
n00b
Joined: 17 Oct 2004
Posts: 16
|
| Posted: Sun Jan 08, 2006 6:11 pm Post subject: Re: HOWTO: set up a local cddb/freedb server |
|
|
| supermihi wrote: |
Filesystem considerations:
Since the database consits of hundreds of thousands of very small files, the filesystem you are using for storing the data has a great influence on how much disk space will actually be used (covering several hundreds of percents from reiser4 to vfat). I *highly* recommend to use reiser4, which uses disc space extremely efficiently with small files, or at least reiserfs. Reiser4 is not yet in the official kernel, you'll either need to patch your kernel or use one of the many kernel versions that include reiser4, like the mm-sources, nitro-sources etc.
|
Resurrecting an old thread here, but I used this to put a local freedb server on an old Cobalt Qube 2, and my experience taught me two things:
1. As a practical matter, you can't put the freedb database on an ext3 filesystem. The December 2005 version of the database is something like 1.5 million little text files. I gave up trying to untar the thing onto an ext3 filesystem when it took more than 36 hours without getting halfway through (and this wasn't on the Qube, so it wasn't a matter of the miniscule processing power that system has).
2. If you don't have reiserfs on your system and you don't want to repartition, loopback is your friend. Just as an illustration, I created and mounted a 5 gig file with a reiser3 filesystem, and untarring the same database took about 25 minutes. |
|
| Back to top |
|
|
supermihi
Guru
Joined: 09 Feb 2005
Posts: 348
|
| Posted: Sun Jan 08, 2006 6:18 pm Post subject: |
|
|
Good tip! Thanks, I'll mention that in the tutorial.
_________________
"You may say I'm a dreamer, but I'm not the only one." |
|
| Back to top |
|
|
Gruffi
Apprentice
Joined: 15 Aug 2003
Posts: 209
Location: Antwerpen - Flanders - Belgium
|
| Posted: Sun Nov 15, 2009 4:19 pm Post subject: |
|
|
You can also use squashfs. (emerge squashfs-tools)
The whole database was 11GB on ext3 but compressed with squashfs it was 950MB
| Code: | cd /mnt/storage/cddb
mksquashfs -always-use-fragments -no-duplicates . ../cddb.squashed
rm -r /mnt/storage/cddb/* #deleting the files takes forever
mount -o loop -t squashfs /mnt/storage/cddb.squashed /mnt/storage/cddb |
works just fine, mksquashfs took 80 minutes tho 
And thanks for the how-to! 
_________________
... and we will show Microsoft, that they cannot take whatever they want. And that Free Software is our software! |
|
| Back to top |
|
|
jlpoole
Apprentice
Joined: 01 Nov 2005
Posts: 154
Location: Napa, CA
|
| Posted: Wed Sep 21, 2011 2:59 am Post subject: |
|
|
Here's a gotcha if you're not famliar with xinet: it appears xinet does not tolerate comments that start after column #1. For example, the above FAQ recommended:
| Code: | | server = /usr/local/bin/cddbd # set this to wherever you installed the binary |
I left the comment in (having just copy/pasted the recommend file contents) as is since /usr/local/bin/cddbd was indeed the server executable. Leaving the comment "# set this..." in the file apparently causes xinetd to quietly reject the entry. When I attempted to telnet to the server, as recommended, the connection failed:
| Code: | themis cddbd-1.5.2 # telnet localhost:8880
telnet: could not resolve localhost:8880/telnet: Name or service not known
themis cddbd-1.5.2 #
|
I then modified /etc/xinet.d/cddbp by removing the comment text as follows:
| Code: | themis cddbd-1.5.2 # cat /etc/xinetd.d/cddbp
service cddbp
{
disable = no
socket_type = stream
protocol = tcp
wait = no
user = apache
server = /usr/local/bin/cddbd
}
themis cddbd-1.5.2 #
|
Note: I also replaced root with apache since I wanted the cgi script to work. Using the modified file, telnet successfully worked.
| Code: | themis cddbd-1.5.2 # telnet localhost 8880
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
201 themis CDDBP server v1.5.2PL0 ready at Tue Sep 20 19:39:56 2011
exit
Connection closed by foreign host.
themis cddbd-1.5.2 # |
Lastly, don't even think about unzipping the database (which is now about 800 MBs) thinking you'll copy the unzipped files to their final destination -- I have a pretty powerful laptop and after several hours, canceled the "cp -r ..." and just moved the tar.bz2 to the directory where I wanted the database to reside and unzipped it there. I think I may investigate the suggestion of using the reiserfs on a file (rather than create another partition), but that's for another adventure.
Also, if you are on ext3 and have the database staged, remember when you "updatedb" to exclude the directory, as in:
| Code: | | updatedb -e /usr/local/cddb |
Finally, just for statistics sake, here's the output from my apache session (which worked right after the install):
| Code: | 210 OK, status information follows (until terminating `.')
Server status:
current proto: 1
max proto: 6
interface: http
gets: no
puts: no
updates: no
posting: no
validation: accepted
quotes: no
strip ext: no
secure: yes
current users: 1
max users: 100
Database entries: 3088945
Database entries by category:
newage: 121127
rock: 799682
folk: 250973
data: 70241
jazz: 183896
misc: 967927
classical: 279550
blues: 159920
reggae: 42189
country: 88599
soundtrack: 124841
. |
Yeah! Classical music is alive and well represented, e.g. near 280,000 entries. |
|
| Back to top |
|
|
jlpoole
Apprentice
Joined: 01 Nov 2005
Posts: 154
Location: Napa, CA
|
| Posted: Wed Sep 21, 2011 3:04 am Post subject: |
|
|
| Also, people may wonder why I went to the trouble to set up a local server. Basically, I wanted to be able to create MP3s and wave format files and found abcde to best fit the bill for my needs. Unfortunately, when it tries to connect to the cddb server that everyone else connects to, if it doesn't get the information sought, it just proceeds using generic naming of the files. The failure to get the appropriate naming due to a server being oversubscribed caused me to undertake the trouble of having my own local server so my files would always be suitably named. When I checked the statistics of the main free server, there were 100 connections -- the limit, so I was competing for a limited resource. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Documentation, Tips & Tricks
