Upgrades to IT system make Macon safer

Many busy days lie ahead for Macon’s Information Technology staff, but the nervous nights have been left behind.

In 2010, all of the city’s basic information flowed between the Finance Department in City Hall, Human Resources in the City Hall Annex a block away, and IT’s home in the Terminal Station at the foot of Cherry Street on one L-shaped fiber optic cable.

Had that cable broken at any point, it could have crippled city government: no payroll for more than 1,000 employees, and no electronic communication among offices in different buildings.

“This used to keep me up at night,” said Steve Masteller, the city’s IT director.

Now those buildings are connected by a double loop of cable, meaning that if there’s a break at any point, data can simply flow in the other direction without interruption. What’s more, network administrator Kerry Hatcher reviewed all the “routes” between locations, greatly increasing efficiency by making sure information ran along the shortest path.

That also made it much more reliable by cutting the volume of traffic on any particular path.

“With the old network, it was kind of like everybody standing in an auditorium yelling at each other,” Hatcher said. “The new network is like a few people in an office passing notes.”

Over the Martin Luther King Jr. holiday weekend earlier this year, Masteller and the rest of the IT staff “rebuilt the entire city” -- consolidating much of the city’s basic electronic infrastructure, increasing its capacity from 1 gigabits per second to 4. A gigabit is 1 billion bits of information, or one-eighth of a gigabyte.

IT personnel removed 29 old servers, consolidating their functions in three new ones, Hatcher said.

Clifford Morey, the Systems & Networks Division manager, said replacing servers saved the equivalent of 940 100-watt light bulbs burning 24 hours a day for a year.

That saves the city $62,000 per year in electricity costs alone, not even counting the reduced need for air conditioning in the server room, Hatcher said.

The three new servers are “sisters,” duplicating each others’ functions, Masteller said. Even if two of them conk out, the third can handle almost all city functions on its own.

“You’d have to take all three down to take the city down,” Masteller said.

Now the city’s electronic network is expanding to include the Macon-Bibb County Fire Department administration and the Emergency Management Agency. Also on tap is a VPN -- “virtual private network” -- enabling the IT staff to log into their work systems from anywhere with Internet access, Masteller said.

Edmund Strozier, the IT assistant director, said IT employees now all carry smartphones, which let them get online -- and hence make them able to work -- from anywhere there’s a good enough signal. That makes monitoring the city’s IT functions possible around the clock, instead of just during regular work hours.

With that and the backup capacity, now there’s no danger of the system really going down, he said. That’s a big improvement from when Strozier started working for the city 14 years ago.

“There’s some big changes here, some very positive changes,” he said.

Even if a virus attacked city computers, it now could be confined to a sub-network in one building, unlike the previous “flat” network, Strozier said.

The main focus of the upgrades is redundancy, building the best system possible with the money allocated to IT, Morey said. “Disaster recovery” was the primary concern, and critical systems were backed up accordingly, he said.

“It would take something unforeseeably catastrophic to bring this system down,” Morey said. City information is automatically backed up in five separate locations, he said.

Those are all “live” hard drives, not tapes, Masteller said, so restoring the system would be almost immediate.

Old ways change

When Masteller came to work for the city about two years ago, the police department and the rest of the city were on separate networks. Now Hatcher has moved all police precincts onto the city network, but there are a few elements of police administration remaining to be moved, Hatcher said.

Likewise, fire stations used to be isolated “islands” but now have Internet access, Masteller said.

This year Information Technology was spun off into its own internal service fund, separate from the General Fund budget, to allow for better cost accounting. Each city department is charged for the computers and service it receives. That $1.2 million is divided fairly evenly across the city except for the Police Department, which makes up 51 percent of IT use, according to a breakdown from the city’s Finance Department.

Though a tight budget has kept city offices from replacing their computer terminals for several years, in this budget year IT will begin distributing new ones, Masteller said. That had to wait until the fiber optic cable and servers were in place. Masteller likened it to driving brand-new cars on narrow, rutted roads. That wouldn’t have been an improvement, so the “road” -- the basic infrastructure -- had to be fixed before city staff got real benefits from new “cars” -- their desktop and laptop computers.

Macon’s IT employees are very competitive, a trait Masteller encourages. They often attack each others’ proposed solutions. If those ideas stand up to criticism from all sides, they’re probably sound, so that only makes the results stronger, he said.

Information Technology workers across the field used to worry mostly about how to run their systems “now,” but that’s changing, Morey said. Today in Macon and elsewhere the focus is shifting to “what-if,” planning for emergencies and growth, he said.

The road ahead

In 16 months, however, there’s a big bump in the road: the impending consolidation of Macon and Bibb County governments, approved by voters July 31 and due to happen Jan. 1, 2014. That means merging not just offices but very different computer systems.

The city and county will have to jointly consider their different approaches to electronic infrastructure and “best practices” in operations, said Steve Layson, Bibb County’s chief administrative officer and interim Information and Technology director.

“With all that said, we don’t want to get out in front of the advisory committee,” he said, referring to the newly named committee that will suggest policies for the combined government.

“Basically it’s just discussion at this point,” Layson said.

Infrastructure may not be much of a problem, he said.

“We’ve got some things that are definitely interchangeable,” Layson said. “Those (systems) are pretty compatible, for the most part. I think you’re going to find the programming, the software end of things, is where you’re going to have some problems.”

That will mean not only creating a new system, but also running both old ones side-by-side for a while until the integration is obviously working, he said.

Masteller didn’t directly disagree with Layson’s assessment, but he said infrastructure is likely to be a considerable issue. The key is communication, he and Morey said. Morey said the city and county do have most of the equipment needed to make connections work, but it will take lots of coordination.

On the legislative level, there’s caution about too much advance decision-making. Layson said the county has $3 million budgeted for a new enterprise resource planning system, which enables information entered in one office to be automatically entered in other office systems, rather than having to be entered time and again in different formats. But an Aug. 21 talk between Grant Faulkner, the county’s Information and Technology Services assistant director, and county commissioners left open whether that should be done now -- or left for the new government.

On Aug. 23, the city and county IT departments got together to start working out their incompatibilities. Though until now they’ve worked independently, pursuing different strategic plans, the first meeting went well, Masteller said. Teams have been assigned to work out solutions to known conflicts, he said. And he talked with Faulkner about coordinating budgets on projects they know both will need.

Meanwhile, the city is pushing ahead on other improvements. Morey said coming soon is a “mesh network,” hooking every city building to every other city building.

“Instead of one big ring, it’ll look like a spiderweb,” he said. That boosts not only reliability but also speed, which will let other city employees be more productive, Morey said.

One new feature will aid in city-county communication: a direct link between the two that can carry 10 gigabits per second.

“Ten gigabits would be 100 floppy discs per second,” Morey said. That basic communication is a prerequisite for working out larger-scale collaboration, he said.

For Macon and Bibb County consolidation to succeed, merging IT functions is crucial, Masteller said. If computers don’t work, nothing else will.

“Departments have to share information,” he said. “Information drives decision-making.”

Telegraph writer Phillip Ramati contributed to this report. To contact writer Jim Gaines, call 744-4489.