Outlook for mythtv-0.22

[depontius]

MythTV-0.22 has been out for about a month now, and is ~arch in Gentoo. At the same time, on the mailing list, it's clear that it's having some growing pains, and it's not clear that even though it's "released", it's ready to move from mythtv-0.21 yet. There is also the Gentoo wish to remove QT3 from portage, yet mythtv-0.21 requires is, so that would be a driving factor to move mythtv-0.22 to "stable."

I'm looking at a family deployment, and am not sure if I want to push out 0.21, or wait until 0.22, so that they never see the version change. At the same time, it would be even worse to give them unstable code.

Is there a prognosis for when MythtTV-0.22 will be considered stable by Gentoo?

[yngwin]

February 21st at the latest, as that is when we will mask Qt3.

[robnotts]

Have been running mythtv-0.22 ever since the vdpau code went into it to get the best out of my nvidia card. Is very stable. There was a blip with qt-4.6.0 where it would sometimes lock, but with qt-4.6.1 it has been as solid as a rock.

[depontius]

I've been fiddling with getting a "family-ready" Myth machine ready for about a year now, with a couple of failed "family tests." Our VCR is starting to die, so I'm getting some more motivation to get this going. I have a few major issues:

1 - I'm not happy with my lirc configuration, so far. I want a fast-forward and fast-reverse, just like an appliance VCR or DVD player, plus more. It's not all that hard to do, just a lot of details, a fair amount of digging to be done, and more details to make lirc do it. Making it into an appliance isn't easy.

2 - I take some shows, transcode them to xvid and burn them as ISO9660 on DVDs. Obviously they're not standard DVDs. I can mount them and play them with most any software. But so far that's not inside the Myth frontend, as far as I know how to use it.

3 - I want this to be an appliance, not a computer. Part of that is instant-on. I'd really like to suspend - not hibernate - the machine. But from what I can tell in /proc/acpi/wakeup, the only thing that can wake it from S3 are USB2/EHCI devices. There's one device that can wake it from S5 (completely off) and everything else wakes it from S4 - hibernation. So far every infrared device I've plugged in gets grabbed by OHCI, meaning it can only wake from S4. I've tried blacklisting OHCI in order to force EHCI to grab the USB ports, but plugged-in devices then demand-load OHCI, and I'm back where I started.

4 - Stability. I have an IMON VFD. Every now and then the whole IMON package goes into la-la land - the VFD starts spouting gibberish and the remote quits working. This state appears to persist through a reboot or regular power off, requiring a full power off (the switch on the back of the PS, or unplug the cord) to clear up. I have an extra (MCEUSB2) IR receiver, and am considering declaring the IMON stuff a loss, even though I spent through the nose for the fancy case with a knob. (The MCEUSB2 IR has the same OHCI/EHCI problem in #3.)

So there are a few issues to work through, some I can solve in software, perhaps some not. Since last fall mostly I fiddle with the machine every month or so, but have been uncertain whether to wait for 0.22 to work on these problems, or work on 0.21 and hope the fixes migrate. I have the machine set up in a different room, and run it when we want to watch something recorded on Myth. But it's not an appliance, yet.

[yngwin]

See also [bug=299222]bug 299222[/bug].

[CoreDump]

Hi,

I have recently upgraded to mythtv0.22, in order to have stable VDPAU implementation (0.21+backport wasn't working right on HD content).

The only trouble was the database issue (utf8 instead of latin1). So it's need some work, but nothing difficult for an experienced mythtv user (it requires database backup/reload and some sql command).

On the stability, I do not have issues (yet). DVB-T HD is now working great.

[tld]

I just posted a comment on that bug explaining that when you've been running MythTV 0.21 with the default Gentoo setup (using utf8) you will need to fix the database using the instructions described here:

http://wiki.mythtv.org/wiki/Fixing_Corr ... e_Encoding

As I mentioned there, it is possible to get this fix done ahead of time (before upgrading) which I did. However if you do that fix beforehand and want to continue to run on 0.21 you must switch your my.cnf to latin1 in order to prevent the "partial corruption" described in the wiki. For that reason it's best to perform the fix only when you're ready to upgrade. After fixing the database and upgrading to 0.22, mysql can safely be left with whatever character set settings you may have.

Be sure to save a pre-fix backup of the database just in case.

As far as MythTV "needing work" on this issue, I'm fairly sure that no change in MythTV could fix the need for doing this...certainly none I've heard of.

I've been busy and wasn't even aware that qt3 was getting masked soon....guess I better get my ass in gear and think about that 0.22 upgrade.

Tom

[tld]

There is one question I have about the MythTV 0.22 upgrade...are there any oddities I may run into when unmasking qt4 and upgrading? Obviously up until now I've had it masked.

My main Gentoo machine has been running qt4 as long as it's been stable and I don't recall anything of note...just wanted to make sure.

Tom

[depontius]

Just saw some activity run across the Gentoo Bugzilla... MythTV-0.22 is going stable, soon. A couple of us have strongly urged that there be a good upgrade guide because of potential problems with the database, and it looks as if that advice will be heeded. There will also be a news item pushed out.

[yngwin]

A couple of us have strongly urged that there be a good upgrade guide because of potential problems with the database, and it looks as if that advice will be heeded. There will also be a news item pushed out.

Only if someone will write it, and the maintainer has indicated he has no time for it.

[depontius]

I just about posted this to the Bugzilla - but felt it's more "forum material."

Code: Select all

(In reply to comment #11)
-snip-
> It is possible to do this ahead of time, which I did (I haven't upgraded to
> 0.22 yet), but that's not the recommended approach.  The reason is that, if you
> fix the database and plan to continue running 0.21 at all, you MUST change your
> my.cnf on BOTH the front and backends to latin1, or you can cause the "partial
> corruption" described in the wiki.  However if you do it when you're ready to
> upgrade, 0.22 will work fine with your mysql configured for utf8.
> 
I've been following this for a while, and this question still confounds me.  "How do I know if I've got partial corruption in my database?"  The only answer I've gotten so far appears to be to do the steps for fixing a uniformly corrupt database and try the upgrade.  If you fail, you've got a partially corrupt database.  I've found no method of detection other than upgrade failure.

There have been at various times Gentoo "suggestions" to fix my.cnf, and once to drop and reload the database, which I've done.  But it seems that utf8 sneaks back into "my.cnf". (There's no ".keep" in there - might that have something to do with it?)  So at this point I don't know what I've got.  I've run 1 backend, several frontends, and even 1 Ubuntu frontend into this system.

This has me scared - and I've gotten nothing better than "Try and see if it works, then start picking up the pieces, and hope."  This sounds like it's going to be a royal pain of backup, try something, restore, try something else - and I fear finally give up, lose 300G of recorded stuff, and start over.

Yes, I'm upset.

I understand that the maintainer doesn't have time to do this. I understand that he's a volunteer.

Let's hold off just a little on marking this stable. I'm on the MythTV mailing list. I'll ask about how the heck you detect a partially corrupt database, and get back here with any response. In the meantime, here's the best guide I've found for actually working with it.
http://wiki.mythtv.org/wiki/Fixing_Corr ... e_Encoding

[tld]

It's an ugly situation for sure. Here's the sort of situation that can be caused by partial corruption as I understand it (one example):

The "people" table in the database has a unique index on the name column. Partial corruption can potentially cause two entries with names that are currently different (from being added using differing character set translations), but which will be the same once the upgrade tries to convert the column from a varchar to a varbinary data type. This change to the table will fail in that situation.

It would be pretty difficult to write something to test for that sort of thing without actually upgrading the database.

I believe that the main things that could cause partial corruption situations are: a) having changed your character set entries in your my.cnf at some point, b) having ported your mythconverg database from another distro that used a different character set settings than mysql in your Gentoo system, and c) doing partial restores of the database.

I'm unclear on that last one (partial restores).

Tom

[depontius]

It would be pretty difficult to write something to test for that sort of thing without actually upgrading the database.

Wouldn't utf8 characters look "escaped", or some such? Might it be a matter of parsing the backup, finding the keys, and then doing some sort of sort? (a little bit of grep, awk, etc.)

I believe that the main things that could cause partial corruption situations are: a) having changed your character set entries in your my.cnf at some point, b) having ported your mythconverg database from another distro that used a different character set settings than mysql in your Gentoo system, and c) doing partial restores of the database.

In my situation, I believe I have 2 risk factors. First is your scenario "a" above. I distinctly remember at some point 2+/- years back being told, either in a forum, by Gentoo Weekly News, or some other news means, to do something about the utf8/latin1 situation. At the time I was told that I should change utf8 to latin1 in my.cnf, then backup, drop, and reload the database. Some time later I noticed that again my.cnf had utf8 in it, so either it wasn't protected from updates, or somehow I missed it. The second risk factor is my daughter's Ubuntu machine is set up as a mythfrontend client, and I have no idea whatsoever what its my.cnf is set as.

I am currently working on accumulating 2 complete series, missing only a few episodes. For one series, I'll get the last 4 episodes Saturday, then I can start transcoding and burning, and it's all done. As for "Eureka", I have no idea when they'll be running again, so I can pick up the missing episodes. But in neither case, nor the other stuff I've got recorded, am I eager to lose it. The "partial restore" was made to sound risky enough that it's next to losing everything, and even then it may not work.

[tld]

I just wrote a perl script that performs the same corruption tests made by mythtv/libs/libmythtv/dbcheck.cpp prior to attempting the schema upgrade. I just submitted it to the mythtv-dev mailing list for some feedback.

My biggest question to them is whether or not these tests are valid or reliable for someone who hasn't yet done the backup/restore fix described in the wiki. That part confuses me.

That aside, it certainly should be a totally safe test. All it's doing is creating temporary tables that are copies of the live tables and performing various tests on those, so there's nothing dangerous going on. I ran it on mine and it passed.

I'll see what they say. Hopefully this will give us some way other that "the hard way" to at least find out if we have a problem.

Tom

[depontius]

I'd be pleased as punch to try out your script. I'd even run it both before and after the dump/drop/restore. As I said, I'll read the raw dump file, if needed.

Incidentally, the other stunt I've pulled from time to time is to take a full "cp -a" of /var/lib/mysql - a complete binary backup. That's in addition to, not instead of, the regular mysqldump.

I've got all of my machines fixed to latin1 now, so theoretically I should be OK doing the suggested drop/dump/restore, even if I'm not ready to upgrade, yet.

[tld]

The script is attached here:

http://www.gossamer-threads.com/lists/mythtv/dev/422959

To see the connection options just run:

perl mythtv_0_22_corruption_test.pl --help

Tom

[tld]

Also, ignore the fact that the syntax described in the help indicates the wrong script name...I'll have to fix that.

Tom

[depontius]

I'll try it tonight at home, first on my as-is database. It may take until the weekend before I can do the drop/reload and do the "after" test.

[depontius]

OK, this is my existing database, no drop/restore, at least not recently. I ran your script against it, as myself:

Code: Select all

Testing people...
Testing recorded...
Testing oldrecorded...
Testing oldprogram...
No failures detected

Next I'll do the drop/restore and rerun your script. Is this the anticipated result? Maybe I've lucked out.
------------------------
Editing rather than another post...

Reading onward in "Fixing Corrupt Database Encoding", it looks as if you take a backup, then pipe it through a sed script to change "SET NAMES utf8" to "SET NAMES latin1". I have none of the former string in my database. What I do have is 83 lines containing "/*!40101 SET character_set_client = utf8 */;" and one line of "/*!40101 SET NAMES latin1 */;" and 83 lines of various flavors of ") ENGINE=MyISAM AUTO_INCREMENT=nnn DEFAULT CHARSET=latin1;" or ") ENGINE=MyISAM DEFAULT CHARSET=latin1;". So I do have mixed utf8 and latin1 stuff in there, assuming that that "/*!40101" thing is some sort of directive, and not a comment. However, it doesn't look as if their de-corrupting script was going to touch the utf8 stuff I've got left. Equally obviously, it would be trivial for me to change it with a slightly different sed script.

????

I probably need to take this on-list, I guess. (Unless someone here understands.)

[tld]

I don't think what you describe is a problem. My backup is the same. I'll elaborate in another post.

However first note that I missed a part of the test that is supposed to be made to the temporary tables in that script. I need to fix that, so your first test may not have been reliable.

Wait until I get a chance to fix that before trying another test. I'll let you know when it's done.

Tom

[tld]

OK...I've fixed that script and posted another reply to my original email to the dev list. Note that it's attached to the third post in that thread (which may not have made it to the archive yet).

As I mention in that post, if it's correct, I for one have problems. I get at least some warnings on all four of the tables tested. Pretty f****** awful.

Regarding you're previous post about your backup I don't think the things you mention are an issue. I have those "/*!40101 SET character_set_client = utf8 */;" entries as well. It appears that it does that before each table create and resets it to whatever it was after. I'll bet that mysqldump does that just to make sure the restore handles something odd like non-ascii characters in the literal table or column names within the script. The "DEFAULT CHARSET=latin1" in the table creates is correct for the MythTV database prior to the upgrade as far as I know.

If there are corruption issues specific to the actual data in table columns...specifically when non-ASCII characters were involved.

It sure looks like I have corruption problems though. I'm going to do the tests manually so I can look at what the actual warnings are that I'm getting. I started doing so on the people table and can see that the warnings I get (229 of the 47306 total rows) appear to be where non-ASCII characters were involved in names. Not fun stuff at all.

Tom

[tld]

Well for one thing, it certainly appears that doing the database backup/restore fix and continuing to run 0.21-fixes (as I did) is a HUGE mistake.

I have a backup of the database immediately before and after I made that fix. The version prior to the fix shows a large number of errors when I use my script to test it. The version immediately after is perfectly clean. However since then, even though my character sets have all been latin1, the database has obviously become a huge mess. I now have a lot of errors.

Unfortunately I did that way back in November. Not fun at all...

Tom

[depontius]

Well thanks for your efforts. I know it doesn't help your database, but it's certainly guiding me on what to do with mine. I pick up the last 4 episodes of "Tru Calling" starting at 8:00 this morning, and I'll get them punched to dvd before upgrading anything. About as soon as I see your upgraded script, I'll be trying it as well.

(I was sick a few years back, sacked out on the couch. "Tru Calling" was about what I could find on TV that day. I have the chance to have the whole series, and save it for another sick day.)

[tld]

It looks like I at least have a plan to fix what I have. A large number of my errors are caused by bad entries in the people and oldprogram tables.

As far as I can see, the oldprogram table is only used for the program search for new shows, which I never use. It generally stores the last 320 days of program titles. I can truncate that table altogether.

Also as far as I can tell in the source, the records in the people table are only referenced in the credits and recordedcredits tables, and most of them are orphaned. Those appear to get added as needed when the program guide gets loaded. I can delete any of those that aren't referenced.

After that, it appears that I'm down to a manageable number of issues that I can fix programatically. Scary shit for sure though.

The newest version of my script is on the dev list archive now. The problem is that you're almost certainly going to show errors unless you perform the backup/restore fix (which you don't want to do before the upgrade). The only safe way to test it would be to make the altered backup as per the wiki and to restore it a different database name (by using mysql manually rather than the mythtv restore script) and then to run my script against that. That would go something like this using the corrected mythconverg-to_uncorrupt.sql backup as per the wiki and a new test database named mythconverg_test. Obviously replace <your_user> and <your_password> respectively. Also note that in the mysql commands only there is in fact no space between the -p and the password:

Code: Select all

mysql -u <your_user> -p<your_password> -e "CREATE DATABASE IF NOT EXISTS mythconverg_test CHARACTER SET latin1 COLLATE latin1_swedish_ci"

mysql -u <your_user> -p<your_password> mythconverg_test < mythconverg-to_uncorrupt.sql

perl mythtv_0_22_corruption_test.pl -d mythconverg_test -u <your_user> -p <your_password>
Testing people...
Testing recorded...
Testing oldrecorded...
Testing oldprogram...
No failures detected

The backup I had from just after the fix came up clean doing that. However as I mentioned, running on 0.21 since then has apparently made quite the mess.

I still haven't received any feedback from the mythtv lists regarding any of this.

Tom

[tld]

Oh...and before you try that, make sure you have enough disk for that second copy of the database. Also, when your done you can get rid of the test database with:

Code: Select all

mysql -u <your_user> -p<your_password> -e "DROP DATABASE mythconverg_test"

Tom