Maintaining a module, killing off another one
I was granted ownership of the jar module. Today, I resumed my quest to kill off the barely limping stopwatch module. Together with nuking STANDALONE mode in jar stuff, I will have landed 75KB worth of -ve diffs this month. It feels so good to delete code.
IO Report
Currently I am focusing on application IO (excluding libraries and IO caused by libraries).
From my empirical measurements, opening individual files on a 7200RPM hard drive costs around 0-40ms. This is on Linux. I presume files open quickly when they are located near previously opened files and slower if a full disk seek is required for them. Combining files is usually a significant win in terms of throughput. It turns out that even warm starts and reading from SSDs can benefit from combined IO. Currently small file throughput ranges from <1KB/s to <200KB/s for files < 500K. Combining files into memory mapped jars bumps that up to 1-1.5MB/s (currently jar files are relatively small, making them responsible for a higher proportion of IO should boost that further).
The biggest gains are to be had on Windows Mobile where almost every seemingly trivial filesystem operation takes 2-3ms.
I would like to reduce the number of files read on startup to a dozen or so to be able to crank up disk throughput. Unfortunately, there is a lot to be done, I could use a great deal of help.
Below is a long list of files gathered by stracing firefox-bin, and what I know about them: /home/taras/builds/minefield/dist/bin/application.ini /home/taras/builds/minefield/dist/bin/browserconfig.properties /home/taras/builds/minefield/dist/bin/platform.ini Haven’t investigated these yet. Perhaps, instead of reading these files should be compiling this info into the xulrunner launcher app?
Update: ted says application.ini used to be compiled in: bug 383167 /home/taras/builds/minefield/dist/bin/plugins
I’m ignoring plugins at the moment. /home/taras/builds/minefield/dist/bin/chrome /home/taras/builds/minefield/dist/bin/chrome/browser.jar /home/taras/builds/minefield/dist/bin/chrome/browser.manifest /home/taras/builds/minefield/dist/bin/chrome/comm.manifest /home/taras/builds/minefield/dist/bin/chrome/en-US.jar /home/taras/builds/minefield/dist/bin/chrome/en-US.manifest /home/taras/builds/minefield/dist/bin/chrome/pageloader.manifest /home/taras/builds/minefield/dist/bin/chrome/pippki.manifest /home/taras/builds/minefield/dist/bin/chrome/reftest.manifest /home/taras/builds/minefield/dist/bin/chrome/toolkit.jar /home/taras/builds/minefield/dist/bin/chrome/toolkit.manifest /home/taras/builds/minefield/dist/bin/chrome/xslt-qa.manifest
The .manifest files describe how to find stuff in chrome jars. Parsing .manifest files is inefficient due to disk seeks. Additionally, current .manifest parsing code appears to be slow on mobile devices. bug 506392 I would like to move .manifest files within jars(and look for manifests within jars when a manifest isn’t found alongside the .jar in the filesystem)
.jar files are a little bit of a cpu hog when it comes to parse the zip file index. This can be done lazily: bug 511754 /home/taras/builds/minefield/dist/bin/chrome/icons/default/default16.png /home/taras/builds/minefield/dist/bin/chrome/icons/default/default32.png /home/taras/builds/minefield/dist/bin/chrome/icons/default/default48.png These icons are parsed by gnome libs 3 times during startup. I wonder if we are initializing some gnome thing 3 times?
/home/taras/builds/minefield/dist/bin/components /home/taras/builds/minefield/dist/bin/components/aboutCertError.js /home/taras/builds/minefield/dist/bin/components/aboutPrivateBrowsing.js /home/taras/builds/minefield/dist/bin/components/aboutRights.js /home/taras/builds/minefield/dist/bin/components/aboutRobots.js /home/taras/builds/minefield/dist/bin/components/aboutSessionRestore.js /home/taras/builds/minefield/dist/bin/components/FeedConverter.js /home/taras/builds/minefield/dist/bin/components/FeedWriter.js /home/taras/builds/minefield/dist/bin/components/fuelApplication.js /home/taras/builds/minefield/dist/bin/components/httpd.js /home/taras/builds/minefield/dist/bin/components/NetworkGeolocationProvider.js /home/taras/builds/minefield/dist/bin/components/nsAddonRepository.js /home/taras/builds/minefield/dist/bin/components/nsBadCertHandler.js /home/taras/builds/minefield/dist/bin/components/nsBlocklistService.js /home/taras/builds/minefield/dist/bin/components/nsBrowserContentHandler.js /home/taras/builds/minefield/dist/bin/components/nsBrowserGlue.js /home/taras/builds/minefield/dist/bin/components/nsContentDispatchChooser.js /home/taras/builds/minefield/dist/bin/components/nsContentPrefService.js /home/taras/builds/minefield/dist/bin/components/nsDefaultCLH.js /home/taras/builds/minefield/dist/bin/components/nsDownloadManagerUI.js /home/taras/builds/minefield/dist/bin/components/nsExtensionManager.js /home/taras/builds/minefield/dist/bin/components/nsFilePicker.js /home/taras/builds/minefield/dist/bin/components/nsFormAutoComplete.js /home/taras/builds/minefield/dist/bin/components/nsHandlerService.js /home/taras/builds/minefield/dist/bin/components/nsHelperAppDlg.js /home/taras/builds/minefield/dist/bin/components/nsLivemarkService.js /home/taras/builds/minefield/dist/bin/components/nsLoginInfo.js /home/taras/builds/minefield/dist/bin/components/nsLoginManager.js /home/taras/builds/minefield/dist/bin/components/nsLoginManagerPrompter.js /home/taras/builds/minefield/dist/bin/components/nsMicrosummaryService.js /home/taras/builds/minefield/dist/bin/components/nsPlacesAutoComplete.js /home/taras/builds/minefield/dist/bin/components/nsPlacesDBFlush.js /home/taras/builds/minefield/dist/bin/components/nsPlacesTransactionsService.js /home/taras/builds/minefield/dist/bin/components/nsPrivateBrowsingService.js /home/taras/builds/minefield/dist/bin/components/nsProgressDialog.js /home/taras/builds/minefield/dist/bin/components/nsProxyAutoConfig.js /home/taras/builds/minefield/dist/bin/components/nsSafebrowsingApplication.js /home/taras/builds/minefield/dist/bin/components/nsSample.js /home/taras/builds/minefield/dist/bin/components/nsSearchService.js /home/taras/builds/minefield/dist/bin/components/nsSearchSuggestions.js /home/taras/builds/minefield/dist/bin/components/nsSessionStartup.js /home/taras/builds/minefield/dist/bin/components/nsSessionStore.js /home/taras/builds/minefield/dist/bin/components/nsSetDefaultBrowser.js /home/taras/builds/minefield/dist/bin/components/nsSidebar.js /home/taras/builds/minefield/dist/bin/components/nsTaggingService.js /home/taras/builds/minefield/dist/bin/components/nsTryToClose.js /home/taras/builds/minefield/dist/bin/components/nsUpdateService.js /home/taras/builds/minefield/dist/bin/components/nsUrlClassifierLib.js /home/taras/builds/minefield/dist/bin/components/nsUrlClassifierListManager.js /home/taras/builds/minefield/dist/bin/components/nsURLFormatter.js /home/taras/builds/minefield/dist/bin/components/nsWebHandlerApp.js /home/taras/builds/minefield/dist/bin/components/pluginGlue.js /home/taras/builds/minefield/dist/bin/components/reftest-cmdline.js /home/taras/builds/minefield/dist/bin/components/storage-Legacy.js /home/taras/builds/minefield/dist/bin/components/storage-mozStorage.js /home/taras/builds/minefield/dist/bin/components/tp-cmdline.js /home/taras/builds/minefield/dist/bin/components/txEXSLTRegExFunctions.js /home/taras/builds/minefield/dist/bin/components/WebContentConverter.js Above files are only read in during the “slow” startup and are “fastloaded” afterward. However, they are stat()ed to make sure the fastload stuff isn’t stale. This is bad: bug 511761. Making these depend on .autoreg seems reasonable.
/home/taras/builds/minefield/dist/bin/extensions This probably should depend on .autoreg too
/home/taras/builds/minefield/dist/bin/updates /home/taras/builds/minefield/dist/bin/updates/0/update.test /home/taras/builds/minefield/dist/bin/update.test This too?
/home/taras/builds/minefield/dist/bin/greprefs /home/taras/builds/minefield/dist/bin/greprefs/all.js /home/taras/builds/minefield/dist/bin/greprefs/security-prefs.js /home/taras/builds/minefield/dist/bin/greprefs/xpinstall.js Reading a directory full of pref files that never change is crazy. bug 507288 provides 10-100x throughput improvement in reading gre prefs.
/home/taras/builds/minefield/dist/bin/defaults/pref /home/taras/builds/minefield/dist/bin/defaults/pref/channel-prefs.js /home/taras/builds/minefield/dist/bin/defaults/pref/firefox-branding.js /home/taras/builds/minefield/dist/bin/defaults/pref/firefox.js /home/taras/builds/minefield/dist/bin/defaults/pref/firefox-l10n.js /home/taras/builds/minefield/dist/bin/defaults/pref/reporter.js This is similar to gre pref situation, but worse because there are more pref files. This is also harder to refactor. I propose having 2 hardcoded pref names for xulrunner applications: app.js burried in a jar file speed and l10n.js for localization convenience.
/home/taras/builds/minefield/dist/bin/dictionaries /home/taras/builds/minefield/dist/bin/dictionaries/en-US.aff /home/taras/builds/minefield/dist/bin/dictionaries/en-US.dic For some reason only one of my computers is reading the spellcheck dictionary at startup, but it is costing me 70ms. I wonder if these can be moved to a locale jar.
/home/taras/builds/minefield/dist/bin/modules/distribution.js /home/taras/builds/minefield/dist/bin/modules/DownloadLastDir.jsm /home/taras/builds/minefield/dist/bin/modules/ISO8601DateUtils.jsm /home/taras/builds/minefield/dist/bin/modules/NetUtil.jsm /home/taras/builds/minefield/dist/bin/modules/utils.js /home/taras/builds/minefield/dist/bin/modules/XPCOMUtils.jsm These modules should be moved into toolkit.jar: bug 509755
/home/taras/builds/minefield/dist/bin/res/broken-image.png /home/taras/builds/minefield/dist/bin/res/charsetalias.properties /home/taras/builds/minefield/dist/bin/res/charsetData.properties /home/taras/builds/minefield/dist/bin/res/forms.css /home/taras/builds/minefield/dist/bin/res/hiddenWindow.html /home/taras/builds/minefield/dist/bin/res/html.css /home/taras/builds/minefield/dist/bin/res/loading-image.png /home/taras/builds/minefield/dist/bin/res/quirk.css /home/taras/builds/minefield/dist/bin/res/ua.css
These need to be in a jar too: bug 508421 /home/taras/builds/minefield/dist/bin/searchplugins /home/taras/builds/minefield/dist/bin/searchplugins/amazondotcom.xml /home/taras/builds/minefield/dist/bin/searchplugins/answers.xml /home/taras/builds/minefield/dist/bin/searchplugins/creativecommons.xml /home/taras/builds/minefield/dist/bin/searchplugins/eBay.xml /home/taras/builds/minefield/dist/bin/searchplugins/google.xml /home/taras/builds/minefield/dist/bin/searchplugins/wikipedia.xml /home/taras/builds/minefield/dist/bin/searchplugins/yahoo.xml I think these should be in a jar too.
This concludes the list of Firefox application files read at startup. I haven’t touched the stuff in the profile directories or libraries. I think it is realistic to get a tenfold reduction in physical files read by Firefox 3.6.
https://bugzilla.mozilla.org/show_bug.cgi?id=383167