Automated Content

We use web scrapers and free data APIs to pull in the data we want to use for our stories.

Our scripts then format the data, fix CP style, determine geographic location and filter data based on the geographic location.

The software generates the story text and writes the headlines, subehaders and cutlines and picks the appropriate photo. It then sends the stories to Blox for publishing.

Almost all our infrastructure is now on AWS. We have one series left still using Snowflake and the old data team resources.

Beach Water Quality

This series is nine stories that run hourly during the day from June to August, seven days a week. The stories all use webscrapers or reverse engineered, undocumented API endpoints to pull in the latest water quality data.

The stories are published to the appropriate daily and community sites and are based on the following regions: Durham, Halton, Hamilton, Niagara, Ontario Beaches, Ottawa, Toronto, Waterloo and York

See an example

Ontario Highway closures

This series runs every day. It pulls data from a reverse engineered Ontario 511 API, cleans it and stores it in our own database. The software writes 13 stories which are sent to the proper websites.

The following regions are the basis for the stories: Durham, Halton, Hamilton, Muskoka, Niagara, Ottawa, Peel, Mississauga, Peterborough, Simcoe, Toronto, Waterloo, York

See an example

Toronto Food Inspections

A story that runs daily, four days per week. It connects to the Toronto Open Data Portal and searches for any new restaurant inspections that were not passes.

The story lists the violators and explains the violations.

See an example

Today's forecast

A daily series that uses Environment Canada data to create a local forecast story. It includes today's weather, a forecast for the rest of the week and historical data.

The stories are published across all our sites first thing every morning. We generate forecasts for the following locations: Barrie, Brampton, Caledon, Cambridge, Cobourg, Guelph, Halton Hills, Hamilton, Kitchener-Waterloo, Markham, Mississauga, Muskoka, Niagara Falls, North Bay, Oakville, Orangeville, Oshawa, Parry Sound, Peterborough, St. Catharines, Welland, Windsor and Toronto

See an example

Auto thefts

This series uses the data from the Toronto Police Service to create a weekly round up of auto thefts. One story is written for each of the six area of Toronto: East York, Etobicoke, North York, Scarborough, Toronto, and York.

These stories run every Thursday.

See an example

Break and Enters

This series uses the data from the Toronto Police Service to create a weekly round up of break ins at homes and businesses. One story is written for each of the six area of Toronto: East York, Etobicoke, North York, Scarborough, Toronto, and York.

These stories run on Tuesday afternoon.

See an example

Food Inspection Roundup

This story combines all the previous food inspection stories during the week into a weekly roundup.

The story runs every Friday.

See an example

GTA Regional Real Estate

This series uses Toronto Regional Real Estate Board data to write stories that compares the average home price on a month-to-month, annual and decennial basis. It also drills down into the specific kinds of housing and gives context around total sales.

We create 36 stories based on the regions covered by TRREB. The 36 regions are: Ajax, Aurora, Bradford West Gwillimbury, Brock, Brampton, Burlington, Caledon, Clarington, Dufferin County, Durham Region, East Gwillimbury, Essa, Georgina, GTA, Halton Region, Halton Hills, Innisfil, King, Milton, Mississauga, Markham, Newmarket, Oakville, Oshawa, Peel Region, Pickering, Richmond Hill, Simcoe County, Scugog, New Tecumseth, Toronto, Uxbridge, Vaughan, Whitby, Whitchurch-Stouffville, York Region.

See an example

Toronto Neighbourhood Real Estate

This series does a deep dive on the different regions around the city of Toronto. Each area includes information from the regional stories, but for every neighbourhood in the region.

The regions for these stories are Etobicoke, Scarborough, North York and Toronto's West End, East End, Midtown and Downtown.

See an example

Provincial Elections

We created an election preview graphic and wrote a results story for every riding.

The Automated Content election coverage was worth more than 150,000 page views, 124 stories, 10 per cent of all election related traffic on thestar.com and generated 10 new subscriber signups from the story pages.

The stories included were automatically updated when a race was called. They included a custom image, the results and context about the riding.

We also were the only media outlet with a story about Haldimand—Norfolk, which caused a bit of a traffic spike when Bobbi Ann Brady, an independent, won the seat.

See an example

Municipal Elections

A major success, we created an automated story for every ward in the city. The stories included the City Council race, and the TDSB, TCDSB, Conseil scolaire Viamonde and Conseil scolaire catholique MonAvenir board elections.

We maximized our success with incredible SEO on these articles. The automated stories were the top Google result for every ward name search on election night. We did so well, all the related stories were links to our other automated content results.

The stories generated more than three times the traffic from the provincial election, exceeding expectations. Christine Loureiro told us, "It's been a sleepy election with low voter turnout. We were expecting to match the average story performance vs [the] Ontario [election], but this is a delight."

See an example

Federal Elections

We created a riding research tool for our reporters.

We will explore how the highway closures series is created.

Overview:

1. Download and clean the data
2. Store the data in our database
3. Convert the data for a specific story
4. Write the story
5. Send the story to Blox

The data for highway closures comes from 511 data. We connect and download the data with the following code:

                                                                        
async function downloadData(start, batchSize) {
    const baseUrl = 'https://511on.ca/List/GetData/Roadwork';

    const queryParams = {
        columns: [
            { data: null, name: '' },
            { name: 'roadwayName' },
            { name: 'description' },
            { data: 3, name: '' },
            { name: 'direction', s: true },
            { name: 'endTime' },
            { name: 'startTime' },
            { name: 'lastUpdated' },
            { name: 'id', s: true },
            { name: 'comment' },
            { name: 'region' },
            { data: 11, name: '' }
        ],
        order: [{ column: 6, dir: 'desc' }],
        start: start,
        length: batchSize,
        search: { value: '' }
    };

    try {
        const response = await axios.get(baseUrl, {
            params: { query: JSON.stringify(queryParams), lang: 'en-US' }
        });

        return response.data;
    } catch (error) {
        console.error('Error downloading data:', error.message);
        return null;
    }
}

async function downloadAllData(batchSize) {
    let allData = [];
    let start = 0;
    let totalRecords = Infinity;

    while (start < totalRecords) {
        const batchData = await downloadData(start, batchSize);

        if (!batchData || batchData.data.length === 0) {
            // Stop if no more data or an error occurred
            break;
        }

        allData = allData.concat(batchData.data);
        totalRecords = batchData.recordsTotal;

        start += batchData.data.length;
        console.log(`Downloaded ${start} records out of ${totalRecords}`);
    }

    return allData;
}
                                    

We then clean the data and add in the geolocation.

 
exports.handler = async (event, context) => {

const batchSize = 100;

// Step 1: Retrieve existing data from DynamoDB
const existingData = await retrieveExistingData();

console.log('Existing records:', existingData.length);

// Step 2: Download new data
const newData = await downloadAllData(batchSize);

console.log('New records downloaded:', newData.length);

// Step 2.5: Clean and combine map locations for newly downloaded data
const mapIconsData = await cleanAndCombineMapLocations();

// Update the output data with map locations for newly downloaded data
const updatedNewData = await updateOutputWithMapLocations(newData, mapIconsData);

// Step 3: Compare data and identify changes
const { itemsToAdd, itemsToRemove, itemsToUpdate } = compareData(existingData, updatedNewData);

// Step 4: Clean and save data
const cleanedDataToAdd = await cleanAndSaveData(itemsToAdd);
const cleanedDataToUpdate = await cleanAndSaveData(itemsToUpdate);

// Step 5: Add cities to data
const dataWithCitiesToAdd = await addCitiesToData(cleanedDataToAdd);
const dataWithCitiesToUpdate = await addCitiesToData(cleanedDataToUpdate);

// Step 6: Clean text
const dataWithCleanedTextToAdd = await fixCleanData(dataWithCitiesToAdd);
const dataWithCleanedTextToUpdate = await fixCleanData(dataWithCitiesToUpdate);

// Step 7: Clean titles
const dataWithCleanedTitlesToAdd = await fixCleanTitles(dataWithCleanedTextToAdd);
const dataWithCleanedTitlesToUpdate = await fixCleanTitles(dataWithCleanedTextToUpdate);

// Step 8: Update DynamoDB
await removeOutdatedData(itemsToRemove);
await addNewData(dataWithCleanedTextToAdd);
await updateExistingData(dataWithCleanedTextToUpdate);
                                    
}

Here is how we clean the data. For example, if we were to send https://mbk0tu83o7.execute-api.us-east-1.amazonaws.com/feb7/ontario-roads/text?inputText=St%20Agnes to the cleaner, it would return St. Agnes. Or if we sent https://mbk0tu83o7.execute-api.us-east-1.amazonaws.com/feb7/ontario-roads/text?inputText=HWY12 it would return Highway 12

async function fixCleanData(cleanedUpdatedOutputData) {
    const apiRequests = [];

    for (const outputEntry of cleanedUpdatedOutputData) {
        const description = outputEntry.description;
        const apiUrl = 'https://mbk0tu83o7.execute-api.us-east-1.amazonaws.com/feb7/ontario-roads/text?inputText=' + description;
        const apiRequest = fetchDataWithRetry(apiUrl, 3, 1000) // Retry 3 times with a 1-second delay
            .then(response => {
                if (response !== null) {
                    const cleaned = response.formattedText;
                    if (cleaned !== null) {
                        outputEntry.description = cleaned;
                    } else {
                        // Mark the entry for removal
                        outputEntry.remove = true;
                    }
                } else {
                    // Mark the entry for removal in case of repeated failures
                    outputEntry.remove = true;
                }
            })
            .catch(error => {
                console.error('Error fetching API data:', error.message);
                return;
            });

        apiRequests.push(apiRequest);
    }

    await Promise.all(apiRequests);

    // Remove objects with remove flag set
    const filteredData = cleanedUpdatedOutputData.filter(entry => !entry.remove);

    return filteredData;
}

We now take the latitude and longitude information and use it to determine which city the closure falls into. At this point we might skip the data if it is not in our coverage area.

For example, if we sent https://mbk0tu83o7.execute-api.us-east-1.amazonaws.com/feb7/ontario-roads/locations-updated?lat=43.6532&long=-79.3832 to our API, it would return TORONTO

async function addCitiesToData(cleanedUpdatedOutputData) {
    const apiRequests = [];

    for (const outputEntry of cleanedUpdatedOutputData) {
        const location = outputEntry.location;
        const apiUrl = `https://mbk0tu83o7.execute-api.us-east-1.amazonaws.com/feb7/ontario-roads/locations-updated?lat=${location[0]}&long=${location[1]}`;

        const apiRequest = fetchDataWithRetry(apiUrl, 3, 1000) // Retry 3 times with a 1-second delay
            .then(response => {
                if (response !== null) {
                    const cities = response.properties.cities;
                    if (cities !== null) {
                        outputEntry.cities = cities;
                    } else {
                        // Mark the entry for removal
                        outputEntry.remove = true;
                    }
                } else {
                    // Mark the entry for removal in case of repeated failures
                    outputEntry.remove = true;
                }
            })
            .catch(error => {
                console.error('Error fetching API data:', error.message);
            });

        apiRequests.push(apiRequest);
    }

    await Promise.all(apiRequests);

    // Remove objects with remove flag set
    const filteredData = cleanedUpdatedOutputData.filter(entry => !entry.remove);

    return filteredData;
}    

We then update our database. This adds in the new data, updates exisiting data and removes the outdated data.

async function removeOutdatedData(itemsToRemove) {

    const deletePromises = itemsToRemove.map(item => {
        const params = {
            TableName: tableName,
            Key: { id: item.id }, // Assuming 'id' is the primary key
        };
        return docClient.delete(params).promise();
    });

    try {
        await Promise.all(deletePromises);
    } catch (error) {
        console.error('Error removing outdated data:', error);
        throw error;
    }
}

async function addNewData(itemsToAdd) {

    const putPromises = itemsToAdd.map(item => {
        const params = {
            TableName: tableName,
            Item: item,
        };
        return docClient.put(params).promise();
    });

    try {
        await Promise.all(putPromises);
    } catch (error) {
        console.error('Error adding new data:', error);
        throw error;
    }
}

async function updateExistingData(itemsToUpdate) {

    const updatePromises = itemsToUpdate.map(item => {
        const params = {
            TableName: tableName,
            Key: { id: item.id }, // Assuming 'id' is the primary key
            UpdateExpression: 'SET #startTime = :startTime, #endTime = :endTime, #lastUpdated = :lastUpdated, #type = :type, #layerName = :layerName, #roadwayName = :roadwayName, #description = :description, #region = :region, #direction = :direction, #comment = :comment, #location = :location, #cities = :cities',
            ExpressionAttributeNames: {
                '#startTime': 'startTime',
                '#endTime': 'endTime',
                '#lastUpdated': 'lastUpdated',
                '#type': 'type',
                '#layerName': 'layerName',
                '#roadwayName': 'roadwayName',
                '#description': 'description',
                '#region': 'region',
                '#direction': 'direction',
                '#comment': 'comment',
                '#location': 'location',
                '#cities': 'cities',
            },
            ExpressionAttributeValues: {
                ':startTime': item.startTime,
                ':endTime': item.endTime,
                ':lastUpdated': item.lastUpdated,
                ':type': item.type,
                ':layerName': item.layerName,
                ':roadwayName': item.roadwayName,
                ':description': item.description,
                ':region': item.region,
                ':direction': item.direction,
                ':comment': item.comment,
                ':location': item.location,
                ':cities': item.cities,
            },
        };
        return docClient.update(params).promise();
    });

    try {
        await Promise.all(updatePromises);
    } catch (error) {
        console.error('Error updating existing data:', error);
        throw error;
    }
}    

We now convert the data into a usable format and generate the JSON required for the BLOX webservice API.

exports.handler = async (event, context) => {
    let yorkData = [];
    let serverProduction = event.testing;

    const splitDataToRegions = async (jsonData) => {
    let dataByCity = {
        york: yorkData,
    };

    jsonData.Items.forEach((obj) => {
        const city = obj.cities.toLowerCase();
        if (dataByCity.hasOwnProperty(city)) {
        dataByCity[city].push(obj);
        } else {
        //console.log(`Unknown city: ${city}`);
        }
    });
    return true;
    };


    const streetList = async (data) => {
    let allRoads = [];

    for (let i=0; i {
        return self.indexOf(value) === index;
    });

    uniqueArray.sort((a, b) => {
        return a.localeCompare(b);
    });

    return uniqueArray.join(', ');
    };


    const articleCodeGenerator = async () => {
    let date = new Date();
    let month = String(date.getMonth() + 1).padStart(2, "0");
    let day = String(date.getDate()).padStart(2, "0");
    let year = date.getFullYear();
    let output = "ONTARIO-" + day + month + year + "-ROADCLOSURE";
    return output;
    };

    const articleHeaderGenerator = async () => {
    let date = new Date();
 
    // Get various date components
    var month = months[date.getMonth()];
    var day = date.getDate();
    var hour = date.getHours();
    let year = date.getFullYear();
    let output = month + " " + day + ", " + year;
    return output;
    };

    const prepareData = async (data) => {
    let output = [];
    let regionCode = data[0].cities;
    let articleCode = await articleCodeGenerator();
    let articleType = "ROADCLOSURE";
    let articleSource = "https://511on.ca/list/events/traffic";
    let municipality = "MTO";
    let streets = await streetList(data);

    for (let i = 0; i < data.length; i++) {
        let item = {};

        item.regioncode = regionCode;
        item.articlecode = articleCode;
        item.articletype = articleType;
        item.article_source = articleSource;
        item.municipality = municipality;
        item.street = data[i].roadwayName;
        item.description = data[i].description.replace(/^"(.*)"$/, "$1");
        item.date_from = await timeConverter(data[i].startTime, "date_from");
        item.date_to = await timeConverter(data[i].endTime, "date_to");
        item.date_from_time = await timeConverter(
        data[i].startTime,
        "date_from_time"
        );
        item.date_to_time = await timeConverter(data[i].endTime, "date_to_time");
        item.reason = data[i].type;
        item.heading_date = await articleHeaderGenerator();
        item.street_total = data.length;
        item.street_list = streets;
        output[i] = item;
    }
    return output;
    };

    const sendToGenerator = async (
    dataSend,
    serverDestination,
    serverProduction
    ) => {
    const data = JSON.stringify(dataSend);
    const apiUrl = new URL(
        "https://a7uwbek5y2.execute-api.us-east-1.amazonaws.com/production/highway-closures-generator"
    );

We then send the story to the story generator. It adds in the metadata, sets the headline and footers and sends the story to Blox

    
"use strict";
//import request from 'request';
const request = require("request");
const bloxMiddlewareAPI = process.env.BLOX_MIDDLEWARE_API_PATH;
const middlewareAPIKey = process.env.MIDDLEWARE_API_KEY;
var RegionName = "";
exports.handler = async (event) => {
    if (event) {
    let serverDestination = event.headers.sento;
    let testing = event.headers.testing;
    let inputPost = event.body;
    let slack = true;
    let theSeries = "Highway Closures";
    let output = [];

    output = await storyWriter(inputPost, serverDestination);

    await publish(
        output,
        RegionName,
        serverDestination,
        testing,
        slack,
        theSeries
    );

    let responseCode = 200;

    let responseBody = {
        message: "success",
    };

    let response = {
        statusCode: responseCode,
        headers: {
        "x-custom-header": "lucas made this",
        },
        body: JSON.stringify(responseBody),
    };

    return response;
    } else {
    let responseCode = 400;

    let responseBody = {
        message: "failure",
    };

    let response = {
        statusCode: responseCode,
        headers: {
        "x-custom-header": "lucas made this",
        },
        body: JSON.stringify(responseBody),
    };

    return response;
    }
};

////////////////////
// Townnews helper functions
function generateID(RegionName) {
    // Generates a unique story id
    // Format: "Series Name DDMMYYYYHHMM"
    // Change the seriesID to the series name
    let seriesID = "AC-Ontario-Highway-Closures-" + RegionName;
    let date_ob = new Date();
    date_ob = date_ob.toLocaleString("en-US", { timeZone: "America/Toronto" });
    date_ob = date_ob.split(/[ .:;?!~,`"&|()<>{}\[\]\r/\\]+/);
    let date = ("0" + date_ob[1]).slice(-2);
    let month = ("0" + date_ob[0]).slice(-2);
    let year = date_ob[2];
    let AMPM = date_ob[6];
    if (AMPM == "PM") {
    AMPM = "p.m.";
    } else {
    AMPM = "a.m.";
    }
    let id = seriesID + "-" + date + month + year;
    return id;
}

////////////////////
// Helper functions
function dateFixer(dateStr) {
    let then = new Date(dateStr).toLocaleDateString("en-us", {
    month: "long",
    day: "numeric",
    });
    return then;
}

function numberFormatter(number) {
    // Fixes CP style for intigers under ten
    // Accepts an int and returns a string
    switch (number) {
    case 0:
        return "zero";
    case 1:
        return "one";
    case 2:
        return "two";
    case 3:
        return "three";
    case 4:
        return "four";
    case 5:
        return "five";
    case 6:
        return "six";
    case 7:
        return "seven";
    case 8:
        return "eight";
    case 9:
        return "nine";
    default:
        return number;
    }
}

function monthFix(month) {
    // Fixes CP style for Month names
    // Accepts a string and returns a string
    month = month.replace("January", "Jan.");
    month = month.replace("February", "Feb.");
    month = month.replace("August", "Aug.");
    month = month.replace("September", "Sept.");
    month = month.replace("October", "Oct.");
    month = month.replace("November", "Nov.");
    month = month.replace("December", "Dec.");
    return month;
}

function generateJSON(
    summary,
    headerContent,
    bodyContent,
    RegionName,
    regionalSites,
    storyKeywords,
    theGeolocations,
    serverDestination,
    theFooter,
    thesections,
    originDomain
) {
    // TownNews API JSON payload set up variables

    function sectionDecider(serverDestination, thesections) {
    if (serverDestination == "star") {
        return ["news/gta/highway-closures"];
    } else {
        return thesections;
        // return ["news"];
    }
    }

    let possibleImages = ["ac-highwayclosures-1", "ac-highwayclosures-2"];

    let outputJson = {};

    outputJson.source_app = "editorial";
    outputJson.published = true;
    outputJson.type = "article";
    outputJson.subheadline = null;
    outputJson.hammer = null;
    outputJson.kicker = null;
    outputJson.archive_time = null;
    outputJson.delete_time = null;
    outputJson.prologue = null;
    outputJson.summary = summary;
    outputJson.source = "AC Bot"; // Need to figure out what we should make this
    outputJson.origins = []; // Not sure
    outputJson.custom = []; // not sure
    outputJson.publications = {}; // not sure
    outputJson.access = "default";
    outputJson.geo = theGeolocations;
    if (serverDestination == "star") {
    outputJson.sites = [];
    } else {
    outputJson.sites = regionalSites;
    }
    outputJson.no_comments = true;
    outputJson.no_reactions = true;
    outputJson.presentation = null;
    outputJson.location = {
    address: "100 Queen St W",
    municipality: "Toronto",
    region: "Ontario",
    postal_code: "M5H 2N2",
    country: "Canada",
    latitude: 43.6529798,
    longitude: -79.3850915,
    };
    outputJson.article_type = "News";

    // TownNews API JSON payload variables that need to be generated
    outputJson.id = generateID(RegionName); // needs to be generated and unique
    outputJson.canonical_url = ""; // might need to be generated
    outputJson.title = headerContent; // needs to be generated headline
    outputJson.social_title = headerContent; // needs to be generated
    outputJson.social_summary = summary; // needs to be generated
    outputJson.print_headline = ""; // needs to be generated
    outputJson.content = bodyContent; // needs to be generated (HTML)
    outputJson.tagline = theFooter; // needs to be generated (HTML)
    outputJson.start_time = new Date(new Date().toString()).toISOString(); // needs to be generated in this format: YYYY-MM-DDTHH:MM:SSZ
    outputJson.seo_title = headerContent; // Need to genertate and update this for SEO
    outputJson.seo_description = summary; // Need to genertate and update this for SEO
    outputJson.origin_domain = originDomain;
    outputJson.url_title = ""; // Need to genertate
    outputJson.related_links = {
    //"Highway Closures": "https://www.thestar.com/news/highway-closures.html",
    //"Open Data Team": "https://www.thestar.com/authors.torstar_open_data_team.html"
    }; // An object with our related links, changes per story
    outputJson.flags = ["web_only"]; // Need to figure out what we should make this
    outputJson.authors = ["opendata@torstar.ca"];
    outputJson.byline = null;
    outputJson.keywords = storyKeywords;
    outputJson.sections = sectionDecider(serverDestination, thesections);
    outputJson.social_twitter_card = "";

    outputJson.relationships = [
    {
        type: "child",
        is_internal: false,
        app: "editorial",
        id: possibleImages[Math.floor(Math.random() * possibleImages.length)],
    },
    ];
    return [outputJson, RegionName];
}

async function storyWriter(closures, serverDestination) {
    let date_ob = new Date();
    date_ob = date_ob.toLocaleString("en-US", { timeZone: "America/Toronto" });

    date_ob = date_ob.split(/[ .:;?!~,`"&|()<>{}\[\]\r/\\]+/);

    let hours = date_ob[3];
    let AMPM = date_ob[6];
    let day = ("0" + date_ob[1]).slice(-2);
    let month = ("0" + date_ob[0]).slice(-2);
    let year = date_ob[2];
    let theDate = day + month + year;

    closures = JSON.parse(closures);

    let RegionCode = closures[0].regioncode;

    let subeheadLedes = [
    "Avoid traffic jams before they happen",
    "Avoid traffic jams before they occur",
    "Avoid gridlock before it happens",
    "Avoid gridlock before it occurs",
    "Expect delays along these routes",
    "Expect delays along the following routes",
    "Expect delays, especially during busy travel times",
    "Expect delays, especially during heavy traffic periods",
    "Expect delays or adjust your route",
    "Expect delays or plan an alternative route",
    "Plan your route before you're on the road",
    "Plan your route before you're in your car",
    "Plan your trip before you're on the road",
    "Plan your trip before you're in your car",
    "Plan your commute before you're on the road",
    "Plan your commute before you're in your car",
    "You shouldn't have to sit in traffic",
    "You don't need to sit in traffic",
    "Don't get stuck in traffic",
    ];
    let theFooter = "";
    let regionalSites = [];
    let thesections = [];
    let storyKeywords = [
    "Traffic",
    "Driving",
    "Construction",
    "#feedprovider_ac",
    "#feed_automated",
    "#ac_hc",
    ];
    let theGeolocations = ["ontario"];
    let originDomain = "";

    switch (RegionCode) {
    case "DURHAM":
        RegionName = "Durham";
        theFooter =
        '
Have further to travel? Find scheduled highway closures outside ' +
        RegionName +
        '.
About this story
This story was automatically generated using open data from Ontario 511. The closures are scheduled by the Ministry of Transportation for short-term or emergency repairs and maintenance. The disruptions may be intermittent or ongoing and can change due to weather, emergencies and other factors.
';
        regionalSites = ["durham-region"];
        storyKeywords.push("#ac_hc_durham");
        storyKeywords.push("#ac_hc_durham_" + theDate);
        theGeolocations.push("Durham Region", "Oshawa", "Whitby", "Ajax");
        thesections.push(
        "news",
        "publications/durhamregioncom",
        "ontario-regions/durham-region",
        "ontario-regions/durham-region/news"
        );
        originDomain = "durhamregion.com";
        break;
    case "HALTON":
        RegionName = "Halton";
        theFooter =
        '
Have further to travel? Find scheduled highway closures outside ' +
        RegionName +
        '.
About this story
This story was automatically generated using open data from Ontario 511. The closures are scheduled by the Ministry of Transportation for short-term or emergency repairs and maintenance. The disruptions may be intermittent or ongoing and can change due to weather, emergencies and other factors.
';
        regionalSites = ["inside-halton", "independent-free-press"];
        storyKeywords.push("#ac_hc_halton");
        storyKeywords.push("#ac_hc_halton_" + theDate);
        theGeolocations.push("Halton Region", "Burlington", "Milton", "Oakville");
        thesections.push(
        "news",
        "publications/insidehaltoncom",
        "ontario-regions/halton-region",
        "ontario-regions/halton-region/news"
        );
        originDomain = "insidehalton.com";
        break;
    case "HAMILTON":
        RegionName = "Hamilton";
        theFooter =
        '
Have further to travel? Find scheduled highway closures outside ' +
        RegionName +
        '.
About this story
This story was automatically generated using open data from Ontario 511. The closures are scheduled by the Ministry of Transportation for short-term or emergency repairs and maintenance. The disruptions may be intermittent or ongoing and can change due to weather, emergencies and other factors.
';
        regionalSites = ["the-spec"];
        storyKeywords.push("#ac_hc_hamilton");
        storyKeywords.push("#ac_hc_hamilton_" + theDate);
        theGeolocations.push("Hamilton", "Stoney Creek", "Dundas", "Ancaster");
        thesections.push(
        "news/hamilton-region",
        "publications/hamilton_spectator"
        );
        originDomain = "thespec.com";
        break;
    case "MUSKOKA":
        RegionName = "Muskoka";
        theFooter =
        '
Have further to travel? Find scheduled highway closures outside ' +
        RegionName +
        '.
About this story
This story was automatically generated using open data from Ontario 511. The closures are scheduled by the Ministry of Transportation for short-term or emergency repairs and maintenance. The disruptions may be intermittent or ongoing and can change due to weather, emergencies and other factors.
';
        regionalSites = ["muskoka-region"];
        storyKeywords.push("#ac_hc_muskoka");
        storyKeywords.push("#ac_hc_muskoka_" + theDate);
        theGeolocations.push("Muskoka", "Huntsville");
        thesections.push(
        "news",
        "publications/muskokaregioncom",
        "ontario-regions/muskoka-region",
        "ontario-regions/muskoka-region/news"
        );
        originDomain = "muskokaregion.com";
        break;
    case "NIAGARA":
        RegionName = "Niagara";
        theFooter =
        '
Have further to travel? Find scheduled highway closures outside ' +
        RegionName +
        '.
About this story
This story was automatically generated using open data from Ontario 511. The closures are scheduled by the Ministry of Transportation for short-term or emergency repairs and maintenance. The disruptions may be intermittent or ongoing and can change due to weather, emergencies and other factors.
';
        regionalSites = [
        "st-catharines-standard",
        "niagara-falls-review",
        "niagara-this-week",
        "welland-tribune",
        ];
        storyKeywords.push("#ac_hc_niagara");
        storyKeywords.push("#ac_hc_niagara_" + theDate);
        theGeolocations.push(
        "Niagara Region",
        "Niagara Falls",
        "St. Catharines",
        "Welland"
        );
        thesections.push(
        "news/niagara-region",
        "publications/st_catharines_standard"
        );
        originDomain = "stcatharinesstandard.ca";
        break;
    case "OTTAWA":
        RegionName = "Ottawa Valley";
        theFooter =
        '
Have further to travel? Find scheduled highway closures outside ' +
        RegionName +
        '.
About this story
This story was automatically generated using open data from Ontario 511. The closures are scheduled by the Ministry of Transportation for short-term or emergency repairs and maintenance. The disruptions may be intermittent or ongoing and can change due to weather, emergencies and other factors.
';
        regionalSites = ["ottawa-valley"];
        storyKeywords.push("#ac_hc_ottawa");
        storyKeywords.push("#ac_hc_ottawa_" + theDate);
        theGeolocations.push("Ottawa");
        thesections.push("news");
        originDomain = "insideottawavalley.com";
        break;
    case "PEEL":
        RegionName = "Peel";
        theFooter =
        '
Have further to travel? Find scheduled highway closures outside ' +
        RegionName +
        '.
About this story
This story was automatically generated using open data from Ontario 511. The closures are scheduled by the Ministry of Transportation for short-term or emergency repairs and maintenance. The disruptions may be intermittent or ongoing and can change due to weather, emergencies and other factors.
';
        regionalSites = [
        "mississauga",
        "brampton-guardian",
        "caledon-enterprise",
        ];
        storyKeywords.push("#ac_hc_peel");
        storyKeywords.push("#ac_hc_peel_" + theDate);
        theGeolocations.push("Peel Region", "Mississauga", "Brampton", "Caledon");
        thesections.push(
        "news",
        "publications/mississauga_news",
        "ontario-communities/mississauga",
        "ontario-communities/mississauga/news"
        );
        originDomain = "mississauga.com";
        break;
    case "PETERBOROUGH":
        RegionName = "Peterborough";
        theFooter =
        '
Have further to travel? Find scheduled highway closures outside ' +
        RegionName +
        '.
About this story
This story was automatically generated using open data from Ontario 511. The closures are scheduled by the Ministry of Transportation for short-term or emergency repairs and maintenance. The disruptions may be intermittent or ongoing and can change due to weather, emergencies and other factors.
';
        regionalSites = ["peterborough-examiner"];
        storyKeywords.push("#ac_hc_peterborough");
        storyKeywords.push("#ac_hc_peterborough_" + theDate);
        theGeolocations.push("Peterborough County", "Peterborough");
        thesections.push(
        "news",
        "publications/peterborough_examiner",
        "news/peterborough-region"
        );
        originDomain = "thepeterboroughexaminer.com";
        break;
    case "SIMCOE":
        RegionName = "Simcoe";
        theFooter =
        '
Have further to travel? Find scheduled highway closures outside ' +
        RegionName +
        '.
About this story
This story was automatically generated using open data from Ontario 511. The closures are scheduled by the Ministry of Transportation for short-term or emergency repairs and maintenance. The disruptions may be intermittent or ongoing and can change due to weather, emergencies and other factors.
';
        regionalSites = ["simcoe"];
        storyKeywords.push("#ac_hc_simcoe");
        storyKeywords.push("#ac_hc_simcoe_" + theDate);
        theGeolocations.push("Simcoe County", "Barrie");
        thesections.push(
        "news",
        "publications/simcoecom",
        "ontario-regions/simcoe-region",
        "ontario-regions/simcoe-region/news"
        );
        originDomain = "simcoe.com";
        break;
    case "TORONTO":
        RegionName = "Toronto";
        theFooter =
        '
Have further to travel? Find scheduled highway closures outside ' +
        RegionName +
        '.
About this story
This story was automatically generated using open data from Ontario 511. The closures are scheduled by the Ministry of Transportation for short-term or emergency repairs and maintenance. The disruptions may be intermittent or ongoing and can change due to weather, emergencies and other factors.
';
        regionalSites = ["toronto"];
        storyKeywords.push("#ac_hc_toronto");
        storyKeywords.push("#ac_hc_toronto_" + theDate);
        theGeolocations.push("Toronto", "Etobicoke", "North York", "Scarborough");
        break;
    case "WATERLOO":
        RegionName = "Waterloo";
        theFooter =
        '
Have further to travel? Find scheduled highway closures outside ' +
        RegionName +
        '.
About this story
This story was automatically generated using open data from Ontario 511. The closures are scheduled by the Ministry of Transportation for short-term or emergency repairs and maintenance. The disruptions may be intermittent or ongoing and can change due to weather, emergencies and other factors.
';
        regionalSites = [
        "the-record",
        "waterloo-chronicle",
        "new-hamburg-independent",
        "cambridge-times",
        ];
        storyKeywords.push("#ac_hc_waterloo");
        storyKeywords.push("#ac_hc_waterloo_" + theDate);
        theGeolocations.push(
        "Waterloo Region",
        "Waterloo",
        "Kitchener",
        "Cambridge"
        );
        thesections.push(
        "news/waterloo-region",
        "publications/waterloo_region_record"
        );
        originDomain = "therecord.com";
        break;
    case "YORK":
        RegionName = "York Region";
        theFooter =
        '
Have further to travel? Find scheduled highway closures outside ' +
        RegionName +
        '.
About this story
This story was automatically generated using open data from Ontario 511. The closures are scheduled by the Ministry of Transportation for short-term or emergency repairs and maintenance. The disruptions may be intermittent or ongoing and can change due to weather, emergencies and other factors.
';
        regionalSites = ["york-region"];
        storyKeywords.push("#ac_hc_york");
        storyKeywords.push("#ac_hc_york_" + theDate);
        theGeolocations.push(
        "York Region",
        "Markham",
        "Vaughan",
        "Richmond Hill"
        );
        thesections.push(
        "news",
        "publications/yorkregioncom",
        "ontario-regions/york-region",
        "ontario-regions/york-region/news"
        );
        originDomain = "yorkregion.com";
        break;
    default:
    // code block
    }

    let content = "";
    // define headingDate, streetTotal
    let headingDate = closures[0].heading_date;
    headingDate = headingDate.split(",");
    headingDate = headingDate[0];
    let headingFullDate = closures[0].heading_date;
    let streetTotal = closures[0].street_total;
    let streetTotalCalc = closures[0].street_list;
    streetTotalCalc = streetTotalCalc.split(",");
    streetTotalCalc = [...new Set(streetTotalCalc)];
    streetTotalCalc = streetTotalCalc.length;

    // split street list then count the length
    let streetList = "";
    if (streetTotal == 1) {
    streetList = closures[0].street;
    } else {
    streetList = closures[0].street_list;
    streetList = streetList.replace(/,(?=[^,]*$)/, " and");
    }
    let summary = streetList;
    if (streetTotalCalc == 1) {
    summary += " is affected";
    } else {
    summary += " are affected";
    }

    let headerContent =
    RegionName + " highway closures for planned roadwork on " + headingDate;

    let random = Math.floor(Math.random() * subeheadLedes.length);
    let lede =
    subeheadLedes[random] + " — " + numberFormatter(parseInt(streetTotal, 10));
    if (streetTotal == 1) {
    lede += " closure is ";
    } else {
    lede += " closures are ";
    }

    lede +=
    "scheduled for roadwork on provincial highways in " +
    RegionName +
    " on " +
    headingFullDate +
    ":";

    content += "
" + lede + "
";
    content += "
";

    for (let i = 0; i < closures.length; i++) {
    let closureData = "
";
    let dateFrom = closures[i].date_from;
    let dateTo = closures[i].date_to;
    closureData += closures[i].description;
    if (dateFrom) {
        closureData += " from " + dateFrom + " ";
    }
    closureData += " " + dateTo;
    closureData += "
";

    content += closureData;
    }
    content += "
";

    content = content.replace(/\s+/g, " ").trim();
    return generateJSON(
    summary,
    headerContent,
    content,
    RegionName,
    regionalSites,
    storyKeywords,
    theGeolocations,
    serverDestination,
    theFooter,
    thesections,
    originDomain
    );
}

// AC-TN-middleware API
async function publish(
    outputJson,
    RegionName,
    serverDestination,
    testing,
    slack,
    theSeries
) {
    return new Promise((resolve, reject) => {
    // CHANGE TO NEW API
    var options = {
        method: "POST",
        url: bloxMiddlewareAPI,
        headers: {
        "x-api-key": middlewareAPIKey,
        sento: serverDestination,
        testing: testing,
        slack: slack,
        region: RegionName,
        series: theSeries,
        },
        body: outputJson,
        json: true,
    };

    request(options, function (error, response) {
        if (error) {
        console.log("It was an error");
        console.log(response);
        throw new Error(error);
        } else {
        let responseID = response;
        //let responseID = JSON.parse(response);
        responseID = responseID.internalid;
        // geturl(responseID, RegionName);
        }
        resolve();
    });
    });
}