Dems score with better data DNC's Linux warehousing project delivered on '50-state strategy' November 15, 2006   E-mail  Printer Friendly  Reprints   Slashdot It!

For example, the DNC outsourced data "cleansing" of state-level voter information in 2004, but got shoddy results, with many records having incorrect phone numbers and lacking multiple addresses that are often necessary to locate voters. State parties that tried to use that data afterwards, in some cases abandoned it altogether because it was unreliable, he said.

Following the DNC playbook for that election year, the group also stopped cleansing data for all but about 18 "swing states," writing off the rest to focus resources where the party felt they mattered most, according to Bickford.

In comparison to 2004, the new system was fast enough to digest updated voter information from all 45 participating states and cleanse it two or three times before Election Day. That meant that state-level operatives got useable data for all 45 participating states, said Bickford and Sullivan.

"The thing I noticed while I was driving around from state to state is that the volunteers were so much happier," said Sullivan. "The difference between phone bank volunteers having, say, 45 percent of the phone numbers accurate versus 70 percent of the phone numbers being accurate is enormous."

Better voter file data from the DNC made a huge difference in the final days of the campaign, said Mark Sullivan, founder of Voter Activation Network (VAN), in Cambridge, Mass., which makes voter file tools that are used by Democrats and other progressive groups.

"There were vast improvements in phone number quality, address quality, and large amounts of consumer data," Sullivan said

While it's unclear whether the DNC data warehouse was a deciding factor in any race, Davis and others cite at least one example of where it came into play: The Florida State Democratic party's efforts to target "Snow Bird" voters from New York and New England states were a direct product of having more detailed voter information with multiple addresses.

"They were able to communicate with Florida voters earlier in the cycle and not wait until people came back in October," said Davis. They could search for people who had addresses in northern states. They never would have been able to do that before," he said.

"[The DNC] put people in areas where we didn't expect big battles to take place, and gave them the tools and [voter] modeling data," said Sullivan. "That played no small part in what happened, especially in some of those states that weren't on the radar."

The DNC database has been a source of controversy in recent months. Following the 2004 debacle, factions within the Democratic Party decided to start their own voter information project, dubbed Data Warehouse and now known as Catalist, under the guidance of former Clinton administration deputy chief of staff Harold Ickes. Ickes was beaten out by Dean to head up the Democratic National Committee.

The Catalist project also bore fruit last week, according to its CTO Vijay Ravindran, who worked for Amazon.com before joining Catalist.

The system, which relies on EMC storage hardware on the back end, was built using open source components such as Linux and MySQL and development frameworks such as Hibernate and Spring. Like the DNC voter file project, Catalist's Data Warehouse is used and is designed to support third-party applications such as VAN, he said.

As for the role Catalist's Data Warehouse played in last week's midterm election, Ravindran said the database of 150 million people was used for a variety of activities including mailings, phone banks, and get-out-the-vote and canvassing efforts in 26 states, affecting around 60 million voters.

Groups using the Catalist data included Emily's List, The Sierra Club, Moveon.org, labor organizations, and America Votes, an umbrella group representing 250 different organizations. Like the DNC, Ravindran pointed to races where the data may have swung outcomes: A group called Women's Voices, Women Vote used modeled data on single, unregistered women in Missouri to target a voter drive in support of Claire McCaskill's successful candidacy against Congressman Jim Talent.

The Sierra Club used the organization's data to target 310,000 infrequent voters who support environmental issues in 33 races across the country, including the successful effort to unseat California Congressman Richard Pombo, Ravindran said.

In general, the system performed well, though Catalist's CTO saw room for improvement.

"Having done this before at Amazon, I know that it doesn't mean anything until you go through your first Christmas. We did scale up for [2006], and we learned how to improve the system for 2007 and 2008," Ravindran said.

In published reports, Ickes has said the Data Warehouse project amounted to a vote of "no confidence" in the DNC effort, and that the newly funded Catalist organization would provide Democrats with voter targeting and data mining capabilities on par with the Republican Party's program (Findit/4689).

Despite the history between powerful figures like Ickes and Dean, Ravindran said that he doesn't pay much attention to the "political dynamic" or history between Catalist and the DNC. He anticipates cooperation in the future.

"We've done the first-tier job of getting [Data Warehouse] off the ground. My fond hope is that the DNC and other progressive groups with valuable data share that data," he said.

Bickford also said he doesn't see the two projects competing with each other. Two voter databases -- one associated with the Democratic Party, the other commercial -- may be fine.

"There are like-minded democratic organizations that will always be extremely happy to have organizations like [Catalist's] Data Warehouse, but would not be able to get information directly from the Democratic Party."

What all those involved with the Democrats get-out-the-vote effort agree on is that the Party has closed the technology gap that previously existed between their party and Republicans.

"The quality of the data has substantially improved. It was a huge step up. And with the overlay data, the state parties can do more, but it's not automated. What we want in the future is to automate that and be able to make intelligent decisions about who to contact," Davis said.

Sullivan agrees, and said that the media underreported the sophistication of the Democratic turnout effort in 2006, and overestimated the abilities of the GOP.

While Democrats now have detailed and accurate enough information to look at sub-areas within individual precincts, they lack the ability to target individuals in the way that Republicans can -- a process called "micro targeting," Davis said.

"The idea that Democrats are doing micro targeting is a myth," Davis said. "If you look at the close races, Republicans were able to do things to narrow the margins, even in this cycle. Their performance was impressive."

However, micro targeting is within reach, and Davis said that the data warehousing solution his company helped develop could work as a platform for it -- for example: using profiles of known voters to match up with other individuals who may be sympathetic, but infrequent poll-goers.

As both Democrats and Republicans reach parity on the technology front, the battle will shift to integrating the various data sources in a seamless manner, said Sullivan.

"We'd like to get the data as soon as its refreshed at DNC, then migrate it into our systems. That would reduce the number of human interventions from what we currently have," he said.

While platforms like Netezza are great for extracting data from huge numbers of records, they aren't well suited to the variety of tasks that enterprise databases perform, Gartner's Feinberg said, noting that Democrats may want to harmonize their competing data warehouse projects.

"The DNC does a lot more than identify voters in Florida or DC, or run a program every two years for an election," he said. "If you have a complete enterprise data warehouse, maybe you can take a subset of that data out and put it on the Netezza box for special functions, like doing targeted campaigns. That way you can run queries against it all day long and not hurt the DNC," he said.

"We're really scratching the surface with what we've done with techniques and technologies," said Catalist's Ravidran. "A decade from now, we're going to look at these first few years where we cleaned up voter lists just so we could do simple queries as the stone age. And they are."

E-mail  Printer Friendly  Reprints   Slashdot It!

TOP NEWS:

COLUMNISTS