Large Datasets in Windows 8: Part 2 – Syncing Datasets to SQLite

Note: This is part two of a two part series. Part one focuses on how to get data from a web service using a C# data layer and display it in a ListView and can be found here.

In my previous post, I showed how to page a ListView by making sequential calls to a web service. Let’s say that the application you’re developing will only have intermittent web access. In this situation, syncing data to a local SQLite database is a good option.

Syncing Data from a Web Service to SQLite

This post uses the same source code as my previous post, so we’ll once again be using the City of Chicago Data Portal, in particular, the Current Employees Names, Salaries, and Position Titles dataset. Since we are only concerning ourselves with the one table, our sync process is going to be very straightforward. We’ll pull down the data in batches, and either insert the row into our database or update the data if it already exists.

If you start to look at the code, you’ll see that the DataSyncService.cs file is the main entry point into the sync process. It is extensible allowing you to sync as many tables as you would like with a fine level of control. We’re going to skip over this class though and move on to the lower level guts of the process contained in DataSyncController.cs. Let’s take a look at the two main functions:

private async Task<bool> SyncTable<T, R>(Func<Task<List<T>>> getItems, Func<R, Task> insertOrUpdateItem)
{
    var items = await getItems.Invoke();
    if (items != null)
    {
        foreach (var item in items)
        {
            await insertOrUpdateItem.Invoke(ModelMapper.MapValues<T, R>(item));
        }
    }

    if (items.Count != BATCH_SIZE)
        return true;

    return false;
}

public async Task SyncEmployeeSalary()
{
    bool done = false;
    int count = 0;
    while (!done)
    {
        done = await SyncTable<ServiceModel.EmployeeSalary, LocalData.LocalModel.EmployeeSalary>(
            new Func<Task<List<EmployeeSalary>>>(() => _remoteDataProvider.GetEmployeeSalariesAsync((count * BATCH_SIZE).ToString(), BATCH_SIZE.ToString())),
            new Func<LocalData.LocalModel.EmployeeSalary, Task>(_localDataRepository.InsertOrUpdateEmployeeSalary));

        count++;
    }
}

SyncEmployeeSalary is called by the DataSyncService and in turn it calls SyncTable. The SyncTable function is a generic function that can be used to sync any table types. The first parameter passed to SyncTable tells it what web service call to make to get a list of objects. The second parameter tells the function what local repository function to call to insert or update SQLite data. Finally, the two types (T and R) provided to the function tell it to map ServiceModel.EmployeeSalary objects to LocalModel.EmployeeSalary objects. This is because the objects we receive from the web service are going to have a slightly different structure than what we store in our database. The while loop inside SyncEmployeeSalary simply batches web service calls. Since the web service has 32,000 rows, it is too much data for us to return all at once so we need to break up our calls. BATCH_SIZE is a local constant set to a default of 500. In your own work you may have to tinker with the number to suit your needs.

In Part 1 of this series, we covered getting data through web service calls, so we’re going to skip straight to the SQLite interaction in LocalDataRepository.cs:

public async Task InsertOrUpdateEmployeeSalary(EmployeeSalary employeeSalary)
{
    var connection = GetConnection();

    var existingEmployee = await (connection.Table<EmployeeSalary>()
        .Where(x => x.id == employeeSalary.id).FirstOrDefaultAsync() as Task<EmployeeSalary>);
    if (existingEmployee != null)
    {
        existingEmployee = ModelMapper.MapLocalModel<EmployeeSalary>(employeeSalary);
        await connection.UpdateAsync(existingEmployee);
    }
    else
    {
        await connection.InsertAsync(employeeSalary);
    }
}

This code is pretty straightforward, we first check to see if a record exists for our current Id. If it does, we update the record. If it doesn’t, we insert it.

In this sample application, I chose to sync data immediately on install and then every 30 days thereafter. The City of Chicago only updates this data quarterly, so syncing more frequently seems unnecessary especially given the time it takes to download 32,000 records. The sync process is kicked off in default.js if needed, and upon completion we navigate to our home page.

Displaying Data from SQLite

Displaying the SQLite data can be done in a very similar fashion to the way we were previously displaying data from the web service. I’ve added a new call to the Application Model, GetEmployeeSalariesLocalAsync, that targets local data instead of remote data. The call still takes in a single page, and since we are able to access SQLite using Linq we can do a Skip(start).Take(length) on the data to perform paging. For this example, this allowed me to give the user the ability to say whether they want to use remote data or local data as they page through the application. When you click the combo box in the lower left corner of the application, a boolean is flipped and the javascript calls the proper get function:

var cmboSelectionChanged = jsUI.eventHandler(function (e) {
    if (e.currentTarget.selectedIndex == 0) {
        isUsingWebService = true;
    } else {
        isUsingWebService = false;
    }
    getTeams();
});

var getTeams = function () {
    if (isUsingWebService) {
        WinJS.Promise.wrap(app.getEmployeeSalariesRemoteAsync(parseFloat(currentPage)).then(employeesLoaded));
    } else {
        WinJS.Promise.wrap(app.getEmployeeSalariesLocalAsync(parseFloat(currentPage)).then(employeesLoaded));
    }
};

Pretty basic. If you want a better explanation how the data is bound, you can find that in my previous post as well.

I tried to write this sync in a way that was extensible and straightforward. That”s why some of the code may seem extraneous in such a simple application but hopefully it gives you ideas of what can be done in more complicated scenarios.

Return to Part 1 – Displaying the Data

Source

Clarity.Samples.LargeDataset.zip

Note: The City of Chicago changed their variable names from last week to this week and I wouldn’t be surprised if they do it again. This may just happen quarterly when they update the data. If you run the project and a javascript error is thrown saying a value cannot be null, it just means the mapping needs to be redone. If you open Fiddler and get values back using the url above and open Clarity.Samples.LargeDataset.Services.RemoteData.ServiceModel.Model.cs you”ll see above the Name, JobTitle, Department, and Salary variables are name mappings. If you replaces those with the corresponding new numbers from Fiddler, everything should work just great again.