Powered By Blogger
Showing posts with label debugging. Show all posts
Showing posts with label debugging. Show all posts

Ok, so for implementing access to a SD Card from boot loader what all we need to implement? Here goes:
  • SD Card block read, identification routines
  • Routines to identify, validate and read the partition information from MBR
  • Routines to perform the FAT (VFAT) data from card.
If you check out the sources code in U-boot, these functionalities has been divide in to the following modules:
  • pxa_mmc.c (/drivers/mmc/)
  • part.c (/disk/)
  • part_dos.c (/disk)
  • fat.c (fs/fat)
  • file.c (fs/fat)
If you directly take the relevant headers and these source files and compile it along E-Boot, of course, you will end up in lot of error, mostly related to
  • data types
  • use of __attribute__
  • structure and macros which are present in Linux headers
Once the errors have been resolved (of course, I am going to leave it up to you), I first enabled and invoked the SD CARD detection routines, I was able to detect the SD CARD, but seemed to failing in reading the MBR, on dumping it, I observed that some bits are not the same as the actual data on SDCARD. So I posted the same on to U-boot mailing list.


Then I figured out that the GPIO configuration was getting overridden in another place in code. Corrected it and there it was working like a charm.

Now I enabled, the FAT access module, here again I was stumped with another problem, the compilation was happening properly, but the Flash Image (Eboot.Nb0), was not generating properly. I.e., I observed that there was no change in behavior if I flash the image, with new functionality, also the time stamp on the file was not updating for NB0 file after compilation.

If I disable the new code added, the Image generation was happening properly. So there I was stuck again with another strange problem. So I went through the code again. After some pain staking and careful analysis, I noticed on thing, there were some array declarations like the following, through out the code:
__u8 get_vfatname_block[MAX_CLUSTSIZE];
The macro MAX_CLUSTSIZE, was having the value 65536 . So it is creating an array of very huge size.

So why didn't it create a problem in BIN file creation, instead the problem is showing up NB0 file creation? Answer is very simple:
BIN files are something similar to a loader, i.e. they will have a predefined record format, and will a piece of loader code to unpack the same. On the other hand, NB0 or ROM Image files are like how the Program will be present in memory, when it is loaded.

What happened was that, when trying to create spaces for all the arrays that had been declared in code, the NB0 file creation was failing (Although it was not shown as a warning or error while building). The size of Rom Image for boot loader is restricted to 256KBytes in Eboot.bib.

Now to work around this problem, I used the free memory in RAM and changed these arrays to pointers and initialized them to some addresses in RAM. Now, I had the module working , with reading of any content in an SD Card.

Now moving on to USB Client Upgrade implementation, I will explain the things later on in the next post

Sometimes it becomes required to extend the existing boot loader, mostly for implementing functionalities like firmware upgrade over a variety of medium like USB Pen Drive, USB Client or even a Micro SD Card. In most of the cases, the dead lines (of course always it will be like it) will be very close too.

For the current project, which I am working now, I had to implement such an upgrade over USB Client (which would be used in production team) and a Micro SD Card based upgrade(which will be used by the end users) in the Windows CE Boot loader(E-Boot).

Previously I was able to port, DAS U-boot and Embedded Linux on the same hardware. Anyone who had worked on U-boot, will definitely know about the advantages and the amount of features that it exports for working on an embedded system hardware. It had the implementation of SD Card access with FAT file system. Now for getting around with the task of firmware upgrade assigned to me, within the time limit, was to make use of the code present in U-boot for MMC Card read operations.

Now that I had chosen what I had to do, I started the analysis of what needs to be done for porting. These were the major difference that I was able to observe between the two systems - Windows CE Boot loader and U-boot:
  • Compilers used for building both of them were completely different. E-Boot was using Microsoft's Cross compiler for ARM. While U-Boot was using GNU C Compiler. This itself is one of the major bottlenecks as the constructs used in one compiler might not be available in the other one. For instance, __attribute__ which was used a lot in U-Boot, was specific to GCC.
  • The source code of U-Boot was tightly coupled with Linux. The headers that were being used, data structure (for maintaining list etc), were taken from Linux source code.
  • The data types (typedefs mostly), available and used were a lot, for the same unsigned char, there were lot of typedefs like uint8_t, u8. Same goes for other types too.
  • There were intelligent mechanisms using macros in U-Boot through various header files for almost every platforms that were available today. That made writing new source code virtually easy.

Then again the objective of my task was to, integrate the module from U-Boot on to E-Boot.
So how do I achieve this? Will explain the details in next post

Monday, November 17, 2008

Remote Tools and CETK over activesync

I will explain on how to connect development pc to a board running windows ce 6.0 os image using Activesync. The development board that I used is similar to the Mainstone III development platform (based on a Marvell PXA 270 processor) and activesync is configured to connect over the USB client port.

Previously when I was trying to connect to my development board using activesync from remote tools, i was getting error message that " Unable to load device side components". The samething happens while I was using CETK. But Sctivesync was connceting properly. Then I started digging into discussion forums for finding a solution to this problem. Here is what I found:

  1. Open C:\Program Files\Common Files\Microsoft Shared\Windows CE Tools\Platman\target\wce600.
  2. Create a folder armV4 and copy the contents of armV4i into this one.
  3. Now Open Kernel Tracker or any remote tools.
  4. Then follow the same steps specified by Sue in her blog post.
Presto!!!... There you have your remote tools working over Activesync.

NOTE: Special Thanks to Sue Loh (Windows CE Base Team Blog) and Yan Sun (CE 6.0 and PlatMan remote tools)

Tuesday, November 11, 2008

Debugging a booting problem

In this post, I will explain about how I debugged a problem in which our development board was taking a lot of time to completly boot up into Windows CE.


The usual bootup time for the development board was around 30-35 seconds, but as we were adding new drivers , the bootup time went up by minutes. The booting was really slow. So I took up this issue.

As i analyzed the booting up process, it was clear that the Boot phase 1 (loading of FMD) was getting over without any hickups. And slow down seems to have been occuring while drivers are getting loaded and initialized.

We can follow two strategies for pin pointing this issue:

Remove each driver one-by-one.
If we are not clear with the source of the problem, then individually eliminating each component one-by-one and observing the behaviour is a straight forward but time consuming way.


Using Celog
In one of my earlier posts, I have mentioned about a cab file that I created with all the celog settings and files. I installed this file on to the development board and configured the registry to launch 'CeLogFlush.exe' during bootup. Also Celog was configured to dump the data into a local file.

[HKEY_LOCAL_MACHINE\init]
"Launch05"="CeLogFlush.exe" -- include celog.dll

[HKEY_LOCAL_MACHINE\System\CeLog]
"Transport"="LocalFile"
"FileName"="celog.clg"
"BufferSize"=dword:100000 // 1MB
"ZoneCE"=dword:815263
"FlushTimeout"=dword:1388 // 5 seconds
"FileFlags"=dword:1 // Keep file open
"ThreadPriority"=dword:C8 // Priority increased to 200
After the complete system was up, I killed the CeLogFlush.exe using ITaskMgr (a really good task manager application) and took the Celog.clg from the device. This output file will contain only the thread IDs, we need to fix this up using readlog (http://devwince.blogspot.com/2008/10/fixing-thread-names-in-celog-output.html).

Now we can view this file in Windows CE Remote Kernel tracker.

Image1


According the Celog data, the booting process went normally till the 7th second since celogflush was started and all the drivers till that point was loaded and threads have been started (that means the XXX_Init routine was completed successfully). But after that the next thread start only at the 52 seconds mark, which was the battery monitor thread. And according the driver loading order in registry, the drivers were the power button monitor and battery driver.

Image2


When I analyzed the XXX_Init of battery driver, I found out that there was a t_printf statement in the init routine. According to the MSDN documentation t_printf will dump the data onto a console, which is again a driver and is not started yet. So when I removed the t_printf statements, the booting speed was back to normal.

:)

Monday, October 13, 2008

Fixing Thread names in CeLog Output

When you are debugging on a standalone system (i.e. which is not connected to platform builder, using corecon), you may need to get the celog CLG file and review the performace data for the session in Kernel Tracker

But the file which you are getting will not have the thread information, it will only have the identifiers for each thread instead of the names. The names of the thread can be acquired using the Readlog utility. First copy the CLG file you obtained from the system into the build release directory and execute the following command :
Readlog -fixthreads celog.clg output.clg

Now output.clg file will have the thread names fixed up.

Fixing Thread Names With Readlog (http://msdn.microsoft.com/en-us/library/aa935935.aspx)

Tuesday, September 16, 2008

When things go wrong....

I found a useful post by Bruce Eitman, regarding how to debug some NOT SO FRIENDLY issues like data aborts etc.

http://geekswithblogs.net/BruceEitman/archive/2008/07/08/summary-of-when-things-go-wrong.aspx

Monday, September 15, 2008

CeLog

Its one of the most helpful tools that I have used in WinCE BSP development. Celog is an event-tracking tool that logs a set of predefined ( I think configured or filtered would be an apt word) for this) CeLog collects the data given by various sources such as applications or drivers in a location configured in registry.

Using tools such as CeLogFlush.exe, we can periodically collect the data and can be analyzed in a development workstation.
A Special Thanks to Sue Loh, for posting a really helpful and detail documentation in her blog in MSDN for using CeLog along with tools such as kernel tracker.
While working on the wrist pc project, we were experiencing some unexpected slow of system while booting up and running applications and we were pondering in dark just to figure out the cause of the snail's pace performance. Then I came across the tutorial given by Sue Loh in her microsoft blog on using CeLog on a standalone machine.

Previously I had used Remote Kernel Tracker, which is built into the platform builder for connecting to a development board using CoreCon and use CeLog to gather Kernel Data (such as interrupt, thread switching, events and other information), on a real time basis (well pretty close to real time). But for this the board needs to be connected to over ethernet or serial to a development work station.

Most of the times problems will arise while testing under real scenarios. So debugging such issues, I came up with a solution. I made a CAB file with the following things:
  • celog.dll
  • celogflush.exe
  • ITaskMgr_arm.exe (A free task manager I found in net, http://urana.info/mobile/wince/itaskmgr/index.html)
  • oscapture.exe
(Also added shortcuts in start menu to launch the exes too, just for the ease of use)

I distributed this cab file among my team and asked them to install in all the boards that we were using. So at any point of the system is showing any slowness, start the 'CeLogFlush.exe' application, which will periodically dump the performance data on to a file. After significant data is gathered, they can use the ITaskMgr_arm.exe to kill the CeLogFlush flush instance and then take the dumped file.

This file can be opened in Kernel tracker to view the information.

In later posts I will explain in detail about CeLogFlush and other tools for performance monitoring.

Regarding how to analyze the performance data, Sue has put up a detailed tutorial in her blog.