My Dream Land Use Reform Package

In the spirit of documenting some of my coursework on this blog, I thought I’d share one of my favorite assignments from this past semester. In my recent U.S. Housing Policy course with former FHA head Carol Galante, we were tasked with developing a mock set of 5 recommendations for addressing the housing affordability crisis through land use reform in the Bay Area. The assignment got me fantasizing a lot about my top 5 policies—no matter how politically feasible. Below, I present three local policies, one state, and one federal policy that are my top 5 land use reform policies for housing affordability. Obviously there are some other high-level reforms (e.g. providing housing as a human right) that I could include here, but sticking with the theme of land use reform is more in my wheelhouse and a more manageable set of possible reforms.

Local Recommendations

At the local level, the rules of the housing development process are largely set in stone by state level policies. However, localities in the Bay Area often make the process more difficult and costly than even state policies demand. Therefore, two of my fantasy local policy recommendations represent common sense reforms (at least among city planners) to make development easier and less costly. The third local reform—land value taxation—is a progressive tax to redistribute the wealth from well-endowed landowners to the general public, who create the social value of the land.

I. More Ministerial Development Approval

For many localities in the Bay Area, the general parameters of the city planning process are largely prescribed by the state. For example, localities must make general plans where they plan which areas of the city can receive future housing growth and zoning decisions must conform to these plans. Furthermore, each locality must review the environmental impacts of housing developments to ensure that they comply with the California Environmental Quality Act (CEQA). This process is already quite extensive and onerous as is. Yet, many localities in the Bay Area make this process even more burdensome by not allowing ministerial approval—that is, approval that is dependent on the judgment of a planning commission—for housing that meets the zoning code, general plan, and environmental standards. Instead, cities like San Francisco mandate discretionary review for nearly every housing development, no matter how small. In Berkeley, neighbors have even appealed a 3-unit development to the city’s Board of Supervisors, who rejected the project even though it met the zoning code.

The state’s Housing Accountability Act prevents cities from rejecting housing developments that do not meet the zoning code. However, many cities enact zoning policies that automatically trigger discretionary review. For example, San Francisco has many zones that automatically trigger “conditional use authorization”, in which the Planning Commission only approves the project if they judge that it “is necessary or desirable to the neighborhood” (SF Planning Department 2017).

This extra planning results in extensive and costly planning review processes for small housing additions or mid-sized apartment buildings that are often more affordable than larger high-rise apartments. Instead, it makes more sense to me to focus limited discretionary planning resources on large apartments that have the biggest impact on the surrounding environment. For whatever limited impacts small to mid-sized developments have on the surrounding neighborhood, it is easy to legislate these into the planning and zoning code. After all, what’s the point of having zoning regulations on paper if we have to individually review every housing development anyway?

Public hearings—like this one in San Francisco—are overused and oftentimes a death knell for housing developments. Image taken from:

Therefore, my first dream policy is to mandate ministerial development approval for small to mid-sized housing projects that meet the zoning code. This simple procedural change could dramatically reduce entitlement costs for developers, increasing housing supply and affordability region wide. In particular, small to mid-sized developments that cost less to build than large apartment buildings could also increase affordability for low to middle income residents.

As always, there would surely be some political pushback for this change. Neighborhood activists often use discretionary review to bring developers to the table to negotiate for community benefits packages. However, these concerns should be limited by the fact that discretionary review would still exist for large projects. In addition to this, some neighbors will dislike losing local control over development decisions. There is no easy solution to this political opposition—emboldened by overly generous planning policies for local residents to begin with. For this reason, it is critical that the state, regional governments, and executive branches of city governments, which are accountable to broader constituencies rather than local neighborhood coalitions, lead the fight for such policies. We’ve already seen some good news on this front in the Bay Area—from Sacramento (e.g. SB 35) to ABAG to aggressive housing production goals by the mayors of the three big Bay Area cities

II. Reduced Parking Requirements

Parking is expensive, costing tens of thousands of dollars and a significant percentage of the cost of constructing a unit of housing. This is especially problematic throughout the Bay Area, where many opportunities for growth are in infill locations where developers must dig underground or build higher to include parking. Yet cities throughout the region still often force developers to build more parking than what the market demands through outdated planning codes that include minimum parking-to-unit ratios.

Moreover, car ownership in the largest cities in the Bay Area, particularly San Francisco, have been declining recently. A recent survey of 80 housing developments through the Bay Area and found that 28% of the parking—adding up to nearly $200 million in costs—went unused (Tranform 2017). These trends suggest that the region can dramatically reduce the cost of housing while at the same time sufficiently meeting parking demand by simply not mandating that developers build parking. Some of the bigger cities in the region—notably San Francisco and Oakland—have already lowered parking minimums around transit. However, there is a lot of room for more reductions, particularly around BART stations in other Bay Area cities, such as downtown Berkeley or Redwood City, both of which have at least one-to-one unit to parking space ratios.

Despite local regulations that mandate parking minimums, many underground parking garages in Bay Area apartments look like this today. Image taken from:

In general, this reform seems to be less politically challenging than ministerial housing approvals, which are a much more radical departure from CA planning norms. In fact, some recent developments in San Francisco have faced opposition from progressive neighborhood groups because parking ratios have been too high. Despite the political momentum behind this reform, there is sure to be some local opposition from neighbors worried about increased neighborhood parking competition. However, if parking competition is fierce enough, there would also be private incentives for developers to build parking with housing, dulling the increased demand.

III. Increased Renter Protections Through a Land Tax

The two reforms above represent a return to free market principles, which can increase regional housing affordability by responding to market demand. However, the free market can also negatively impact existing renters, especially when prices increase dramatically in the short term due to high demand. Therefore, many CA cities must invest in renter protections to protect existing low-income residents. Aside from a moral justification for protecting residents from displacement, we can justify anti-displacement policies in that society values socioeconomic diversity and housing stability as a public good, worthy of government protection. Typically, the best way to provide public goods is for the government to subsidize this good.

Although the government can use any funds to subsidize renter protections, I propose that they tie these funds to the social value of land through a land value tax. Based on the writings of Henry George, a land value tax is levied on the socially created value of land, separate from the value that the owner contributed to the land. As demand increases to live in the Bay Area, the socially created value of the land increases—through no effort of the existing landowners. In this case, it’s necessary to tax the landowners in order to extract that social value and give it back to the public. In this case, I suggest using the funds to mitigate one of the largest negative effects of this growth—displacement of existing residents. For example, these funds could be used to fund existing renter’s protection enforcement, affordable housing acquisition, or rental assistance programs.

Land value enlarged by increases in the social value of land. Should be taxed! Image taken from:

Out of the three local policy recommendations, this is probably the most politically difficult. Landowners would likely be vehemently opposed to any tax increases. However, there is a good argument this is a small compensation to pay after years of benefiting from artificially capped property taxes through Proposition 13. Furthermore, tying the funds to affordable housing and anti-displacement programs would likely be politically popular among liberal constituencies, particularly renters.

IV. State Recommendation: Common Sense CEQA Reform

Localities largely have to play by the state’s rules in planning and zoning decisions. One of the most restrictive and problematic of those rules is the California Environmental Quality Act (CEQA). A by-product of a wave of environmentalism in the 1960s and 70s, CEQA was initially passed with the intent to publicize the environmental impacts of a development and to consider mitigations for negative environmental effects (Fulton and Shigley 2012, p. 180). However, the law gives extraordinary power to opponents of development by allowing anyone to file a lawsuit for any reason, and placing the burden of proof on local governments. This has resulted in decades of CEQA abuse—lawsuits filed by neighbors or labor unions concerned not with environmental impacts but rather with protecting property values or securing union contracts.

In fact, in a review of all CEQA lawsuits filed from 2010-2012, Jennifer Hernandez has found that 2/3 of the housing-related CEQA lawsuits target environmentally friendly infill housing in urban areas (Hernandez, et. al. 2015, p. 63). Once a lawsuit is filed, it can take months or years for a court to reach a decision, during which time the project is stalled. Not only does this time delay supply increases from responding to the market, but it also increases the costs of development so that the project may become infeasible.

Fortunately, there are a number of common sense reforms—summarized well in Jennifer Hernandez’s study—to CEQA litigation that could make a big impact. First, CEQA litigants are able to conceal their identities and reasons for suing a development (Hernandez, et. al. 2015, p. 78). Requiring that litigants both identify themselves and the environmental reason for suing a development would represent an easy change to protect CEQA from non-environmental related abuses. Furthermore, projects can be sued an unlimited amount of times even after a court has upheld the project against previous challenges. Restricting lawsuits after a court has ruled that a project meets CEQA would go a long way towards curtailing abuses. Lastly, many courts that rule in favor of litigants deny housing projects entirely rather than requiring further environmental mitigation or public disclosure, even if the project meets the vast majority of environmental standards. CEQA litigation reform that would allow for more mitigation relief rather than vacating a project altogether would go a long way toward curtailing abuse by litigants that merely want to halt projects for non-environmental reasons. Overall, curbing these abuses would restrict CEQA lawsuits to those truly concerned with environmental impacts, helping lower costs for developing housing statewide and, in particular, in urban areas.

Although these CEQA reform measures make sense, a lot of entrenched interest groups—especially wealthy homeowners and labor unions—rely on abusive litigation to fight against development. The rules surrounding CEQA litigation have been in place for decades and reform disrupting this status quo would surely cause political backlash. However, appealing to the common sense nature of these reforms and targeting abuses of CEQA rather than legitimately environment-related lawsuits could help overcome political opposition. Furthermore, a growing awareness of CEQA abuse is sprouting up among the state’s progressive politicians, supported by pro-development yimbys (“yes in my backyard”).

V. Federal Recommendation: Funding Incentives for Land Use Reform

Lack of housing supply is a large issue in most of the high demand markets nationwide. This lack of supply is, in turn, a result of state and local land use control, where already-existing residents have the private incentive—and the political control—to block additional development. At the beginning of the 20th century, states and localities devised land use controls as a result of power delegated by the federal government. To be more specific, local control over land use is founded on the basis of police power, or the state’s power to regulate to protect the “health, safety, morals, and general welfare” of its inhabitants. This power is given to localities from states, who have this authority on the basis of the 10th amendment, which reserves all government powers not specified for the federal government by the Constitution to the states.

So if land use control authority is local, what can the federal government do? Its role is mainly limited to promoting states to “do the right thin” through example legislation and federal grants. In fact, the Obama administration did just that by proposing $300 million in funding through the Local Housing Policy Grants to regions that supported new zoning and land use regulations to create and expanded, more diverse housing supply (Furman 2015). These were grants proposed to localities to support any new policy—from infrastructure expansion to code writing assistance and implementation—that would reform land use policies to promote development. Although the total amount was a rounding error in the overall federal housing spending, it represents a small but significant step in the right direction. A few million dollars to hire a team dedicated to land use reform might be all a locality needs to make sweeping changes to build more multifamily housing in previously exclusive single-family areas or decrease parking requirements near transit-oriented neighborhoods. In this way, small incentives from the federal government could be enormously impactful to affordability. After all, it was the federal government’s model Standard State Zoning Enabling Act (1922) that contributed to the widespread adoption of restrictive zoning in the first place.

Federal support for land use reform could encounter opposition from Republican members of Congress that are generally less supportive of housing funding. However, the opposition is not guaranteed. This policy would represent a largely market-based approach to housing affordability and barriers to economic growth, which is likely appealing to conservatives.


Bialick, Aaron. 2014. “Car-Free Households Are Booming in San Francisco.” Streetsblog San Francisco (blog). August 16, 2014.

“Conditional Use Authorization.” 2017. San Francisco Planning Department.

Dineen, J. K. 2016. “In a Wealthy SF Neighborhood, Residents Fight Low-Income Housing.” San Francisco Chronicle. November 16, 2016.

Fulton, William, and Paul Shigley. 2012. Guide to California Planning. 4th ed. Solano Press Books.

Furman, Jason. 2015. “Barriers to Shared Growth: The Case of Land Use Regulation and Economic Rents.” The Urban Institute.

Hernandez, Jennifer, David Friedman, and Stephanie DeHerrera. 2015. “In the Name of the Environment.” Holland & Knight, August.

Li, Roland. n.d. “As Millennials Reject Car Ownership, Developers Reduce Parking in Projects – San Francisco Business Times.” Accessed November 8, 2017.

Taylor, Tracey. 2017. “Berkeley’s Bid to Stop New Housing Being Built Overruled by Judge.” Berkeleyside (blog). July 25, 2017.

Transform. 2017. “Comparison Report | GreenTRIP Parking Database.” 2017.

How Long do Housing Developments Take in San Francisco?

This blog post is a shortened version of a longer academic paper. You can find the latest draft of the full paper here.

I. Introduction

Housing developments in San Francisco often endure long and dramatic battles between the public, developers, and the city itself. Yet, for all of the discussion around these battles, there does not exist any comprehensive accounting of just how long the housing development process is. This information is critical for either policy makers considering how to better streamline development review or developers considering the feasibility of a certain site. In this post, I’d like to introduce some preliminary data exploration on development times in San Francisco. Specifically, I have estimated the time it has taken housing developments completed in the recent past (mid 2009 to the first quarter of 2017) to reach completion after submitting their planning or building permit applications.

In order to do this, I appended quarterly datasets known as the development pipeline reports from the San Francisco Planning Department. These datasets are meant to be cross-sectional, but contain other helpful information such as the date a project was first filed. With this information, I transformed the multiple pipeline datasets into one project-level dataset of development timelines in the recent past. In this process, I had to make a few assumptions about development timelines—notably, I assume that a project is completed on the first date of the quarter when it “disappears” (i.e. is not present in) from the pipeline reports after being under a status of “construction” in the previous report. This means that the completion date is accurate within 3 months of the actual completion date. Note that these estimates also only measure the time from official application submission to completion, so do not take into account any preliminary review time, such as preliminary project proposals or community meetings prior to their official application.

Without further ado, I’d like to give a broad overview of what the data says. If you’re interested in the process to create this data, you can find the code on Github here.

II. Housing Development Time Overview

For the purposes of this post, I say development “time to completion” to mean the time from the point of submission of a building or planning entitlement application to the completion of the project. After some data cleaning (filtering out housing developments with duplicate building permits, etc.) we’re left with 1,147 housing developments between Q3 2009 and Q1 2017 that have completed at least one stage of the development process. Of these, 721 reached completion. Figure 1 shows the distribution of these projects by time to completion in years. The median development time for these housing projects is about 3.8 years—meaning over half reached completion within 4 years of submitting applications. Despite this, there are a sizable amount of projects that take more than 10 years—the maximum taking 24 years.

The bunching below 5 years in this histogram is partially because this includes all housing developments without regard to their unit count. As a result, there are many 1-2 unit projects below the 5 year point and many multifamily apartment buildings—the majority of unit additions in the city—above 5. Instead, we can normalize this measure by dividing the project development time by the amount of units added by the development. Instead, Figure 2 below shows a histogram of time to completion for housing developments that included 10 or more new units. These projects have a much greater median time to completion of about 6 years.

In addition to total time to completion, we can measure three separate stages of the development process with this data: 1) the entitlement stage, or the time from submission of planning or building permit application to receipt of building permits 2) what I call the “construction prep stage”, or the time between receipt of planning and building entitlements and the start of construction and 3) the construction stage. Figure 3 contains histograms for the lengths of all three project stages and contains varying sample sizes depending on how many projects contain this information. The figure shows that the entitlement stages tends to be the longest stage and also has the most variation between developments. While most construction prep and construction stages are less than 2 years, entitlement stages are oftentimes much longer—between 2 to 4 years or more.

III. Variation in Development Times by Project Size, Neighborhood

What could explain the variation in times to completion shown in the figures above? It’s easy to think of a few factors, most notably the size of the development. Figure 4 draws a scatter plot of total time to completion against the numbers of units added. There is some but no significant positive correlation between units and development time. In order to explore this further, I break down median time to completion by some common unit-size categories in Figures 5 and 6. As expected, small developments (those with 1-10 units) have the shortest completion times. However, these developments make up the vast majority (about 75%) of housing developments completed over this time period. Furthermore, very large projects (more than 200 units) take the longest at just over 6 years.

Interestingly, projects adding between 50-200 units take less time than those adding between 10-50 units. Perhaps this is because 50-200 unit projects are closer to downtown areas—where neighborhood opposition is less fierce—than 10-50 unit projects. Figure 6 breaks down development time by unit-size categories into the 3 stages of the development process. Again, most variation in review time comes from the entitlement process. Interestingly, the longer entitlement phase for mid-sized 10-50 unit projects makes up for the fact that larger developments take longer in the post-entitlement stages.

Finally, I explore differences between neighborhoods. For Figures 7 and 8 below, I use the 41 neighborhoods created by the Planning Department. Because some neighborhoods only get small housing developments while others only get very large developments, I normalize time to completion by dividing by units added. For these graphs, I also drop neighborhoods that have less than 10 developments in the sample. The results point to a clear pattern: when we divide time to completion by unit size, lower-density neighborhoods—such as Twin Peaks, Bernal Heights, or the Richmond District—outside of the core of the city have both larger total time to completion and entitlement review times. Conversely, most of the highest-density neighborhoods—places that have also seen the most development, such as Soma, Mission Bay, or Hayes Valley—have the shortest review times. Figures 9 and 10 show these results on the map.

These results emphasize an important point: despite much smaller development in outlying neighborhoods, the city spends just as much time reviewing developments in those areas. However, these developments have much less impact on the surrounding environment, and so likely do not merit commensurate review time as large apartment buildings near downtown. These results suggest that the city could have a large impact on housing streamlining by reducing the scrutiny on small to mid-sized developments. This would also likely conserve limited city resources.

Figure 9. Housing Time to Completion by Neighborhood

Figure 10. Housing Entitlement Time by Neighborhood

IV. Conclusion

I hope that this exercise has shown the potential benefit for local governments to collect data on the housing development process. Unfortunately, cities rarely maintain data on housing production at all, let alone the time it takes to create housing. In this example, we have learned not only how long developments take on average, but also how size and location impact review times. Armed with this knowledge, the data suggests some key areas of improvement in the housing review process.

First, both the longest stage of the housing development process and the stage with the most variation is the entitlement review process. This variation suggests that it has the most potential for delay or expedition of developments. Considering this, Mayor Ed Lee’s executive directive to streamline housing review definitely appears to target the correct part of the housing development process. Second, despite being smaller, mid-sized developments in low-density areas appear to have just as long development review times as larger apartment buildings. This signals that the development review process is at least somewhat misaligned, as the city could be saving time reviewing these projects and instead allocating this time to larger, more disruptive projects. Furthermore, this misallocation of priorities likely has a large impact on housing affordability, as mid-sized developments are likely to be more affordable than large apartment buildings, which are more costly and so demand top market rents.

Unfortunately, I cannot say from this analysis exactly why mid-sized developments have the such long review times. However, many of these projects are in low-density areas that are either 1) subject to more neighborhood resistance or 2) are not within specific plan areas, which qualify for streamlined environmental review. Unfortunately, the Mayor’s process streamlining directive does neither of these things, instead focusing speeding up internal procedures without reducing requirements. Disgruntled community members can appeal and delay projects no matter how hardworking and efficient the review process is. Instead, a more politically difficult but effective policy requires reducing regulatory requirements themselves, such as the amount of discretionary review for small projects. For example, the city could consider creating more specific area plans in outlying neighborhoods so that small projects in these areas would be subject to much less environmental review.



Quarter 1 2017 Development Pipeline Release

Time for another update to the San Francisco development pipeline database! After the release of the quarter 1 2017 data earlier this year, we can now visualize the projects completed in the last quarter of 2016. I have linked to the map below and here is a link to the full map on its own. The map allows users to filter to see projects completed in the latest quarter (Q4 2016).

As I have described in earlier posts, this is not a perfect exercise. This data is not really meant to be longitudinal, but I have stitched together multiple cross-sectional releases anyway. I make the assumption that a project is completed the quarter before it stops appearing in the pipeline reports. According to the reports here, this means that the project received its Certificate of Final Completion from the Department of Building Inspection. Most of the time, this seems like an accurate assumption.

Highlights of this release include:


All this adds up to nearly a thousand units added in the eastern half of the city in the final quarter of last year. Happy browsing!

Reflections on the MacArthur Transit Village

The MacArthur Transit Village was prominent in Oakland housing news in the first few months of this year. For those who don’t know, the MacArthur Transit Village is a large multifamily development between the City of Oakland, BART, for-profit and non-profit affordable housing developers on the former site of a parking lot next to the MacArthur BART station. The plans are big: 875 new units (including 146 affordable) with tens of thousands of square feet of non-residential ground-floor space. In general, it’s great news!

I thought now—a few months after the developer received final entitlements—would be a good time to reflect on some of what I learned from the development process. As part of a research methods course in the Masters of City Planning program, a few of my classmates and I wrote a paper about the project, which you can find here.

1. There’s a surprising amount of community support for housing in Oakland

I tend to consider people like me—pro-development urbanist nerds that celebrate density—an outlier. However, that’s not what I saw at the city meetings for the MacArthur Transit Village. If anything, two-thirds of the crowd was supportive of the project and cited the kinds of arguments—from the environmental benefits of transit-oriented housing to the need to increase housing supply at all ends of the market and increased property tax revenue, etc.— that you would hear in a city planning program. The supporters groups were a spectrum ranging from YIMBY activist groups (props to East Bay Forward), labor union members, to longtime area residents. Although these groups certainly have their differences, they each coalesced in support of the project for good reasons. Contrary to what I thought before the meetings, it isn’t just urbanist twitter arguing for smart, transit-oriented infill housing.

A few of my classmates and I wanted to survey neighborhood residents outside of the active community members present at the city meetings. In order to do this, we intercepted approximately 150 community members at intersections around the MacArthur BART station. Although this was in no way a random sampling of neighborhood residents, I think we learned some interesting things. For those who had heard of the project (about half), we asked if they supported or opposed the development. 24 supported the project, while 18 opposed. This sort of evidence, combined with the support I heard at the city meetings, gives me hope for future coalition building around a pro-development housing agenda. YIMBYs certainly leave out some important nuance from the discussion, and the views of other left-leaning housing activists and neighborhood residents could greatly contribute to a fuller, more formidable housing alliance—something that Jordan Fraade and Daniel Kay Hertz have discussed recently.

2. There is a disconnect between community needs and developer exactions

Our surveys showed that there was an apparent disconnect between what residents rated as the greatest community concerns and the exactions that the city is demanding from the transit village developers. As the table below shows, the community seems to clearly identify affordable housing availability as the greatest community problem. However, in the development agreement the city is demanding contributions for parks and other miscellaneous capital improvement projects in exchange for a density bonus. These are improvements that could instead likely come from the general tax revenue additions from the project. Instead, the city could use exactions in the development agreement as contributions to the city’s affordable housing fund or to more below-market rate units in the area. The exactions are not a lot, but every little bit helps. This is something that the city is struggling with in aggregate as well, and Oakland as a whole might be well suited to focusing developer exactions on subsidizing below market rate units.

3. Differences in project awareness is a possible culprit for this disconnect

Another interesting finding from our survey were differences in project awareness among age and housing tenure, shown in the figures below. Also, as Figure 7 below shows, residents who were familiar with the project were far more likely to have heard of the project via informal community associations rather than the city itself. In general, it seems like older homeowners were more likely to be aware of the project, and these residents were most likely to hear about the project from their neighbors or the news rather than the city itself.

Prior to the city’s meetings to decide the fate of the transit village, the only community outreach was in voluntary community meetings hosted by the developer or the local city councilor. Older homeowners tend to be the residents with the most social capital to hear about these meetings and thus, are best positioned to influence the negotiation process with the developer. While this is of course speculation, I’d bet that this same group would be more likely to support expenditures on parks, streets, and other miscellaneous amenities rather than affordable housing. Instead, a more comprehensive neighborhood survey would likely have given a better sense of the true neighborhood priority—housing. While I don’t know the practical solution to this problem, there’s a lesson here for planners about how we reach out to neighborhood groups. In particular, there’s definitely a place for more objective information gathering about neighborhood needs.

Q3-2016 SF Development Data Is Out

A little late on this, but the SF Planning Department released the latest release of its Development Pipeline series sometime over the winter holidays.

I have created a nice map of the residential developments from this data here. When filtering the data, the “All Proposed” data will show the Q3-2016 data. In sum, this is the latest snapshot of development activity happening in the city.

In addition to including all proposed developments, users can filter the data by 7 categories based on the latest status of the project: Under construction, building permit issued, building permit reinstated, building permit approved, building permit filed, planning department application filed, and planning department application filed. In general, the development process proceeds as follows: Planning department approval –> building permit approval –> Construction.

The current snapshot represents roughly what we have seen in the past: a relatively high amount of development in the eastern half of the city, especially around the Market Street corridor. By contrast, the western half of the city has little to no consequential development, aside from the Parkmerced megaproject.

Some logistical notes:

Also included in the map are developments from previous releases that have been completed. Developments leave the pipeline when they are granted a final “certificate of completion” from the Department of Building Inspection. While, in most cases this pinpoints the quarter of completion, it is not a perfect method.

Happy browsing!

Mapping Residential Development in San Francisco

San Francisco’s housing crisis is no secret. As perhaps the biggest city planning issue in the Bay Area, activists from all over the region—neighborhood organizations, planning professionals, NIMBYs, and YIMBYs—have fiercely debated and fought over new housing, development by development. Yet for all of the concerns and passion surrounding the issue, it is surprisingly difficult to find data on where, when, and by how much housing is being produced within the city.

In this long-form post, I try to bring this information to life from the data annals of the SF Planning Department. To preview what’s below, I explore patterns of housing production within San Francisco—where, and by how much has housing increased in the past few years? Next, I combine this data with neighborhood rent patterns from Zillow to explore how this production might relate to housing affordability.


San Francisco has been producing tens of thousands of new jobs while typically only adding between two to four thousand new housing units each year (Moretti 2013). By the laws of supply and demand, this has naturally led to a shortage of affordable housing. This is not just a problem that affects the poor. Some economists have estimated that if metropolitan areas increased housing supply with outside worker demand, U.S. GDP could by as much as 9.7% (Chang-Tai and Moretti 2015).

Governments recognize that this is an issue. As part of the state-mandated Regional Housing Need Allocation (RHNA) process, the Association of Bay Area Governments (ABAG) has targeted San Francisco to provide 28,869 additional units of housing between 2014 and 2022 (ABAG 2014, p. 24). As we will see shortly with the maps of recent production in the past 3 ½ years, current production is roughly on track to achieve this goal. At the beginning of 2014, San Francisco had an estimated 379,597 units, so meeting the RHNA goals would represent about an 8% increase in housing (SFPD 2015, p. 5).

The Development Process

Note that the RHNA goals are not binding. Instead, this just means that San Francisco has to plan to accommodate this much housing by providing favorable zoning and funding to support these goals. Ultimately, meeting housing demand is up to the market. Although the development process is endlessly complex, I will spare my readers the gory details in exchange for some key points to understand the maps below.

There are two parallel permitting tracks: building permitting and planning permitting (entitlement). While the former ensures that a building meets building standards, the latter makes sure that it conforms to the land use zoning regulations. After construction, the Departments of Building Inspection looks over the development one more time and issues a Certificate of Final Completion, which is when I mark developments as completed in the maps below. With all this red tape, the median time from submission of first proposals to completion is typically 3 ½ years (“SFPD 2014, p. 15).

All this is to say that there is no getting around the zoning code. If there is any hope for a variance, developers usually need the support of the community, which typically decries any project out of scale with the ‘character’ of nearby development (Dineen 2016). These facts, combined with the map below of the zoning code, foreshadow a result that we can see in the interactive maps: highly concentrated development in the few places that it is allowed.

Figure 1. – SF Zoning Code 2014
SF Zoning Code 2014.
Notes: This map is a simplified version of the SF zoning code 2014. It shows that the majority of zones in San Francisco are low-density residential (in yellow). By contrast, only a small portion of the city allows for high-density mixed-use development (orange or brown). Source: SF Development Pipeline Report Q3 2014.

Map of Residential Projects

 The interactive map below allows users to click on the dots to see more details about each development. I have also linked the data with the Google Maps Javascript API for street view panoramas. Hopefully this can give users a sense of the neighborhood for each development. Furthermore, the map allows users to switch between recently completed projects and currently proposed projects. The map for recently completed projects shows residential completions in San Francisco from the SF Planning Department’s Development Pipeline Data. This data tracks residential completions between Q4 2012 – Q1 2016 (the only period for which there is consecutive data). The first thing that stands out to me is that the total number of projects seem fairly spread out. Despite this, it is clear that large recent developments have been concentrated in certain areas. In particular, the Market Street Corridor, SoMa, and the Dogpatch appear to have particularly high levels of development.

As for currently proposed developments, these appear to be a bit more dispersed but with the same general trends. Note that there are more dots for the currently proposed data because this includes all projects currently in the development pipeline, no matter how long the time horizon. This data includes all projects that currently have applications on file with either the planning department or the Department of Building Inspection and so they span from planning permits that have just recently been filed to construction. Furthermore, this map assumes that all projects will be brought to completion, which overstates what will actually be built. However, this map is probably a realistic prediction of what future development will look like on average.

Map by Neighborhood

The dot maps suggest that large residential development are concentrated, but in order to get a better sense of this concentration, I have aggregated units built with a polygon map. Because I will shortly examine development’s impact on neighborhood affordability, I use neighborhood boundaries from Zillow for this aggregation.

While not totally unexpected, it is startling to see how concentrated development is. Out of the 13,950 net units added over this time period, about half of these come from one neighborhood: SoMa. Furthermore, the top five neighborhoods—SoMa, the Financial District, Downton, the Western Addition, and Bayview—comprise 11,933 net units, or 86% of that total. By contrast, large areas in the western part of the city, such as Richmond or the Sunset District, have added less than 50 units each.

Although the housing supply is growing with demand, not every area of the city is sharing this burden. It’s important to point out that this is due to deliberate zoning decisions made by the planning department in the past 10 years. However, behind every planning decision is tremendous political pressure from neighborhood associations and the like. We can probably trace this development back to NIMBYism.

Impacts on Neighborhood Affordability

As an example of a research question that can use this data, I have merged the map data with Zillow’s neighborhood-level median rent index.[1] Has increased housing supply helped stem rising rents in neighborhoods with a large amount of residential development? Or could it actually have led to increasing rents? Contrary to basic microeconomic theory, some claim that increased development at a local level can actually increase rents due to positive externalities from property value improvements.

First, I examine data on recent units built between Q4 2012 and Q1 2016 and rent change over that same period in histograms. The first histogram reveals the truly extreme nature of the outliers. First, the figure shows that the overwhelming majority of neighborhoods added less than 200 or so units. Second, there is a small middle group that added between 500 and about 2000 units. Finally, there is SoMa in a class of its own.

Figure 2. – SF Units Added by Neighborhood
Notes: Figure shows amount of neighborhoods by number of units added over the span of Q4 2012 to Q1 2016

The second histogram shows that despite the high concentration of development, rent change is fairly evenly distributed across the neighborhoods. In good news for proponents of economic theory, two of the 5 neighborhoods with the lowest percentage increase in median rent—SoMA and the Western Addition—were also in the top 5 in units produced. However, the other 3—the Twin Peaks, North Beach, and Parkside—all added less than 100 units.

Figure 3. – SF Neighborhood Rent Change
Notes: Figure shows amount of neighborhoods by percentage rent change over the span of Q4 2012 to Q1 2016

To get a more complete sense of how increasing units in the market impacted rent change, I estimate the correlation between net units built and percentage change in rent. In particular, I estimate a line of best fit between these two variables with the ordinary least squares method. The figure below shows this line. The line does have a negative slope, indicating that for every 1000 units produced prevents rent increasing by about 3-5 percentage points. However, it is difficult to trust the negative slope of the line. The result is heavily influenced by one outlier, SoMa, which can be seen in the lower right-hand corner of the plot. The r-squared of the estimate—or the measure of how much of the variance in rent change is explained by units built—is only -.06. This indicates that only approximately 6% of the change in rent can be explained by the amount of units produced. In sum, there does not appear to be any clear relationship between rent change and units produced.

Figure 4. – Rent Change Versus Units Added

Lastly, I examine the relationship between average rent over the period Q4 2012 and Q1 2016 and net units produced. Perhaps wealthy neighborhoods with high property values restrict development through political influences on the zoning process and we can reveal this through correlations. In this case, the lint of best fit is also negatively sloped, indicating that a 1000 units increase in housing is associated with approximately 130 dollars less average rent. However, like the previous figure, it is difficult to give much weight to this result. The dots show that they are essentially uncorrelated with one another. In this instance the r squared is only -.026, indicating that about 3% of the variation in average rent can be explained by the amount of units built.

Figure 5. – Average Rent Versus Units Added

Yet, the fact that there is essentially no correlation in these two graphs is a result in itself. Although housing production has a clear impact on regional housing prices, the local impacts of increased supply are less clear. Getting to the bottom of this question requires data for a longer period, a more fine-grained geographic unit, and a complicated econometric model. There is no clear narrative one way or another, and that’s part of the reason that housing debates are so contentious.

Why Should Planners Care?

America’s cities are, for the most part, rapidly growing. In particularly high-demand areas like the Bay Area, this rapid growth is leading to rapid growing pains. With this development comes the need to build, but local zoning laws often confine the areas where that growth can happen. In fact, as we have seen above, the vast majority of growth can be concentrated in a very small portion of the city. This places a disproportionate share of the burdens of growth—crowded new developments, increased traffic, and tall new buildings—in certain neighborhoods.

It is no accident that neighborhoods like the Mission have been the poster child for gentrification in San Francisco. Yes, young professionals are attracted to the neighborhood because of its attractions, but the Mission is also one of the only places where new development is allowed. It’s time to shift some of the blame from gentrifying millennials to local planning decisions.

Furthermore, rapidly growing neighborhoods may not receive the benefits of increased housing supply with rent decreases. Although regions are clearly better of with increased housing supply on the whole, this makes little difference if the local benefits of development are ambiguous. Control over land use is local, and if cities don’t feel the benefits on the neighborhood-level, then urban areas will have a difficult time accommodating population growth with increased housing. Of course, we’ve already seen this with NIMBYs all across the U.S.

Moving Forward

I’m a firm believer that the first step in solving any issue is information gathering and sharing. That’s what I intend to do with this project and this blog moving forward. For tracking San Francisco development, I have set up a list of issues and future goals on the project’s github repository here. Items on the agenda include increasing user interactivity, increasing the time frame, exploring how current development relates to the existing housing stock, and, ultimately, gathering this data for other cities.

The housing debate starts with an honest, open dialogue and such a dialogue necessitates comprehensive and easy-to-access information. I hope that this project is a first step toward achieving that goal. Interactive maps of housing production might be useful not just for planners and developers, but also for neighborhood associations or residents interested in the broader context of regional housing development.


“2014 San Francsico Housing Inventory.” 2015. San Francisco Planning Department.

Chang-Tai, Hsieh, and Enrico Moretti. 2015. “Why Do Cities Matter? Local Growth and Aggregate Growth.” NBER Working Paper, no. No. 21154 (May).

Dineen, J. K. 2016. “In a Wealthy SF Neighborhood, Residents Fight Low-Income Housing.” San Francisco Chronicle. November 16.

Moretti, Enrico. 2013. “How to Make SF Housing More Affordable.” SFGate.

“Regional Housing Need Plan: San Francisco Bay Area 2014-2022.” 2014. Association of Bay Area Governments.

“San Francisco Pipeline Report: Quarter 3 2014.” 2014. San Francsico Planning Department.

[1] In particular, I use the Zillow Rent Index (ZRI) Multifamily, SFR, Condo/Co-op time series found here.

Residential Additions in SF Concentrated in a Handful of Areas

Continuing the recent line of posts about SF residential development, I’ve finally decided to aggregate residential completions over the past 3 1/2 years rather than look at present development. While not surprising, the results are stark: SF housing additions are extremely concentrated in a handful of areas, particularly the South of Market area.

I have also used this post to experiment with the geopandas package in python. I am a complete novice and ran out of patience for some of the legends for these graphs, so apologies for their quirks. Hoping to update this post when I’m more experienced!

Onto the data…first, let’s plot dots of completed construction projects in the past 3 1/2 years. In particular, these are residential completions in the SF development pipeline from Q4 2012 – Q1 2016 because this is the only period for which there is consecutive data. The graph shows what we’ve seen in my previous posts of recent development: spread out, but with much larger developments on the eastern side of the city and, in particular, along the Market Street corridor. In anticipation for my next two maps, I overlay this data with neighborhood boundaries for San Francisco from Zillow. That one blue dot in Bayview has negative 1 units added. I sadly have not yet found a good way to integrate it into the legend.

Residential Development at the Project Level
Residential Development by Net Units Added

Next, the moment we’ve (read: I’ve) been waiting for… let’s aggregate by neighborhood! For those interested, I do this by performing a spatial join in geopandas between the two layers (both projected into UTM zone 10).

Net Units Added by Neighborhood
Net Units Added by Neighborhood

While not totally unexpected, the map is startling. Pretty much all of the residential additions in the past 3 1/2 years are on the eastern half of the city, and SoMa has added more than 3 times the amount of units as the nearest area. Moreover, areas like Richmond and the Sunset District have added less than 50 units each! Although the housing supply is growing with demand, not every area of the city is sharing this burden. It’s important to point out that this is due to deliberate zoning decisions made by the planning department in the past 10 years. However, behind every planning decision is tremendous political pressure from neighborhood associations and the like. This development pattern can ultimately be traced back to NIMBYs.

Finally, let’s look at affordable units by neighborhood.

Net Affordable Units by Neighborhood
Net Affordable Units by Neighborhood

In general, the same areas building any units are the same ones building affordable units. But look at all the zeros! Clearly affordable housing is not an enticing prospect for most of the city. SoMa, I declare you the winner.

SF 2015 Residential Development

Following up on my post a few weeks ago about the state of current residential development in SF, I decided to focus more narrowly here. In particular, above I have mapped the residential developments that were completed in 2015, broken down by whether a development has affordable units or not. In technical terms, affordable units are defined as those for which rent equals 30% of income of a household with an income at or below 80% of San Francisco’s “fair market rent” (defined by HUD). In other words, they have some sort of rent ceiling. While I do not distinguish by number of units on the map, this information is available for each development by clicking on it.

Surprisingly, there are not many clear spatial patterns here. Development—both market rate and those with affordable units—appear to be fairly uniformly distributed throughout the city. In future posts, I hope to map this development for multiple years to explore the broader context of development.

Data comes from the 2015 Housing Inventory Report by the SF Planning Department.

Mapping SF Residential Development

For all of the talk surrounding the Bay Area’s housing crisis, I have found it difficult to understand not only the scale, but also the geography of the problem. Although we can get a decent sense of where development is happening via Census data on housing, this is at best polygons at the county or MSA level. The housing crisis is a problem at the most local levels of government. While the market would probably supply more housing if allowed, local land use controls at the municipal level have time and again blocked development.

So where can we find data on residential development at the municipal level? This is much harder than I initially thought. Most cities do seem to collect this information in their planning departments, but few have this information publicly available. One of the few planning departments that does publish such data is San Francisco’s. Through what’s known as their development pipeline reports, we can identify and map all development projects in the city that would add residential units or commercial space. This consists of all developments which have at least submitted applications to the planning department or department of building inspection. Furthermore, this data excludes projects that do not add residential units or commercial square footage (e.g. roofing upgrades). Within this data, we can group projects by different stages in the development process, such as planning review, building permit review, or construction. Furthermore, we have information on the number of net units added, which can be negative if a project adds less development than the previous residence had.

To start with, let’s take a look at a map of current projects by net units added. This data comes from the development pipeline’s second quarter 2016, which is the latest data available. For all intents and purposes, we can consider this the current state of new housing development in San Francisco.

A few notable points. First, the South of Market (SoMa) area stands out for both the overall quantity of development and the size (in terms of net units) of those developments. This has not gone unnoticed, as some have even called SoMa the “next phase of San Francisco’s urban history.” Second,  the overwhelming majority of new residential development (over 80%) is less than 50 units. Although large luxury apartment buildings in SoMa may grab the headlines, these are definitely outliers in a city that is still mostly zoned for low density. Thirdly, there is a lot of development in low-density areas of the city (e.g. Richmond or the Sunset District) that have relatively restrictive zoning. People don’t seem to shy away from building there even if small developments are all that is allowed. Finally and perhaps most importantly, look how well the map above corresponds to a map of the zoning code! Pretty much all of the big developments seem to be outside of standard residential zoning.

Overall, the above points signal that the market is not the problem. At first glance, residential development seems to be happening fairly uniformly across the city. On the other hand, there is tremendous regulations as to how this development can take place. At least in terms of units per development, zoning is king.


Remember BART’s Twitter fight earlier this year? Weekday exit numbers back them up

As many in the city planning twittersphere may know, the Bay Area Rapid Transit (BART) system’s twitter account has, from time to time, engaged customers in lively debate. In one particular instance earlier this year, BART responded to customer complaints over track maintenance that the system has had an unpredictable rise in ridership coupled with a lack of investment.

Oh snap: Twitter fight
Oh snap: Twitter fight

[ref]Screen shot taken from ([/ref]

Leaving aside the state of BART’s infrastructure and investment, I want to take a closer look at BART’s ridership here. Exactly how much, and in which areas, has BART ridership increased? This simple exercise can help transportation planners determine both which stations serve the most people and where the ridership trends are heading. This, in turn, can serve as a metric to allocate limited resources. While this limited analysis does not fully answer these questions, it exposes some interesting trends to guide the inquiry, along with providing compelling evidence in favor of BART’s twitter defense.

First, let’s look at average weekday exits —defined as a single instance of a rider exiting a station—in 1999. I concentrate on weekday exits here because, anecdotally, workdays seem to generate the most customer frustration. While being late to a Sunday brunch is embarrassing, tardiness to a board meeting is probably worse. Furthermore, weekdays are when ridership is highest. As for 1999, this is the earliest year that I can find from BART’s weekday exit records. However, this works well because it roughly approximates the beginning of the tech boom mentioned in the twitter fight above.

Unsurprisingly, downtown San Francisco stands out as the single outlier on the map. While most stations in 1999 had around 5,000 exits per week, the Embarcadero and Montgomery stations downtown both had over 25,000. Also, the East Bay has fewer exits than the southern outskirts and of San Francisco—emphasizing that this area was the dominant commuting corridor at that time. The only exceptions to this are downtown Berkeley and the two downtown Oakland stations, all three of which had around 10,000 exits per week. This evidence certainly suggests that there was a relatively manageable set of heavily travelled stations. What does the map of 2016 exits look like, you ask?

Ridership grew almost everywhere throughout the system. Traditional heavyweights such as downtown San Francisco, Oakland, and Berkeley grew even bigger, with the top two stations—Embarcadero and Montgomery stations—now accounting for over 45,000 exits per weekday. Perhaps most importantly to the service quality of BART, the outskirts of major cities and suburbs grew sizably (with several new stations appearing). Not only could this put more strain on an aging system, but it also spreads track use over a wider area, making quality of service more and more difficult to maintain.

Finally, what do the numbers say in percentage terms?


Viewed through this lens, downtown San Francisco is far from king of the system. While there is still a wide gap in terms of absolute number of exits, Oakland, Berkeley, and the East Bay suburbs all equal or exceed SF in growth of ridership. Also, the outlying areas of the system are booming! More and more riders are taking longer rides, which surely puts strain on the system. Whoever is in charge of the BART twitter account was not lying. Riders may want to take it easier on a system that, built 60 years ago, has experienced dramatic growth in ridership over such a short period.


Data Sources and Decisions

Average weekday exit data by station comes from BART’s Ridership Reports page, found here. Additionally, geospatial data (kml file format) for the BART system comes from the BART website here. Unfortunately, matching on a string name variable was necessary to merge the geospatial data with the weekday exit data. There is a python script that includes a dictionary for this matching process at my github page here. In sum, I export a csv file with a crosswalk between the two datasets and then merge them within Carto. Also in the python script, I generate the percent change variable used in the above figures.

I made a number of data decisions within Carto during the mapping process. First, I chose a dark basemap and bright colors for the stations for contrast. I also made sure that the basemap’s labels do not overshadow the station points layer. In order to visualize the data, I chose a bubble map to compare an attribute—namely, station exits—of a points layer.

The BART station has expanded over the period 1999-2016, and the BART maps reflect this. I have purposely excluded stations that did not exist in 1999 from the 2016 map. One station, Warm Springs, is due to open in the Fall of 2016 and has been excluded from all maps. Although track expansions after 1999 are present on the 1999 map, I have decided to include the full extent of the present-day BART track system for two reasons: 1) the tracks layer is not split up into expansions, which makes it impossible to drop the expansion portions without dropping the whole system 2) I felt that the tracks were illustrative of the expanse and direction of the present-day BART system, reinforcing my overall point about BART’s enormous growth.

As for the bubble maps, I decided to use equal interval sizes for the bubbles. This is mainly because it easier for the reader to understand. Furthermore, the data in percentage terms is fairly uniformly distributed while the exit maps in absolute numbers have large outliers, which would be lumped together with much smaller stations in some other distribution.

Finally, I decided to label stations because I mention some specific stations in the article. Also, some of my readers might be unfamiliar with the BART system. However, labels would clutter the map at first glance, so I have them only appear at a specified zoom level.