Vision Without Surveillance: Using Zero-Shot CLIP for Scalable Built Environment Feature Extraction

Malikka Rajshahi; Mariela Alfonzo; Corey Ashby; Evan Taylor; Benjamin Feuer

doi:10.66838/sauc.6430

Autores/as

Malikka Rajshahi City University of New York Graduate Center https://orcid.org/0009-0005-1897-2427
Mariela Alfonzo Department of Technology, Culture and Society, NYU Tandon School of Engineering https://orcid.org/0000-0003-0281-4908
Corey Ashby State of Place
Evan Taylor Arizona Center for Algae Technology & Innovation (AzCATI), Arizona State University https://orcid.org/0009-0002-4703-3806
Benjamin Feuer Department of Computer Science, Stanford https://orcid.org/0000-0002-7938-542X

DOI:

https://doi.org/10.66838/sauc.6430

Palabras clave:

Urban Planning, AI for cities, Vision-Language Models, Planning support systems, Urban metrics, Scalable urban feature extraction

Resumen

Municipalities need current, granular built-environment data to support effective planning. However, current collection methods are expensive, challenging to scale, and present privacy concerns. To address this gap, the authors collaborated with the U.S. Department of Transportation (USDOT) through a Small Business Innovation Research (SBIR) initiative. We developed a scalable, privacy-preserving vision-language model (VLM) pipeline that leverages zero-shot CLIP to extract 150+ interpretable urban design features from street-level imagery. Applied to Philadelphia’s bus network, the pipeline processed over two million images and generated block-level predictions for more than 8,000 blocks. This approach enables scalable, evidence-based urban planning across cities.

Descargas

Los datos de descargas todavía no están disponibles.

Estadísticas globales

48 Visualizaciones	34 Descargas
82 Total

Biografía del autor/a

Malikka Rajshahi, City University of New York Graduate Center

Malikka Rajshahi is a Ph.D. student in Computer Science at the CUNY Graduate Center and a machine learning engineer and research collaborator at State of Place. Her research focuses on AI in urban science and machine learning for the public interest, with particular interests in privacy-preserving AI, multimodal learning, and data-driven methods for understanding cities and evaluating urban interventions. She holds a B.S. in Computer Science from New York University’s Tandon School of Engineering.

Mariela Alfonzo, Department of Technology, Culture and Society, NYU Tandon School of Engineering

Bridging the worlds of academia and practice, Dr. Alfonzo is the Founder and CEO of State of Place, an AI-driven urban design data and predictive analytics software that helps citymakers make more effective, cost-efficient, bottom-up, transparent, and accountable citymaking decisions to create more thriving places. State of Place grew out of Urban Imprint, an urban design-behavior research and consulting firm, Dr. Alfonzo established in 2005, via which she worked with a range of private and public sector real estate stakeholders translating data into evidence-based design, development, and planning guidance. In the Fall of 2014, Dr. Alfonzo was recognized as one of Urban Land Institute's 40 under 40 best young land use professionals around the globe. Additionally, she was awarded a Fulbright to examine walkability in China in the Fall of 2013, where she focused on the socio-cultural and policy challenges to implementing walkable development. Dr. Alfonzo is a Research Assistant Professor at New York University, where she's been since 2011, and has taught courses around entrepreneurship, career, and leadership development, as well as urban design. Dr. Alfonzo has a Ph.D. in Urban and Regional Planning with an emphasis on Urban Design and Behavior, as well as a professional Masters in Urban Planning, from the University of California, Irvine (UCI), and a Bachelor of Arts in Architecture and Psychology from the University of Miami.

Corey Ashby, State of Place

Corey Ashby is a software and research engineer with experience in image and video processing, graphics, cloud media systems, and software development. His work has spanned computer vision and perceptual imaging research, GPU-accelerated image processing, video encoding and compression, cloud-based media pipelines, and developer-focused software systems. At State of Place, he contributes software engineering expertise to the development of scalable technologies for extracting and analyzing built-environment data from street-level imagery.

Evan Taylor, Arizona Center for Algae Technology & Innovation (AzCATI), Arizona State University

Evan Taylor advises and mentors early stage startups from vertical farming to performing arts centers, judges business plan competitions, and mentors student entrepreneurs out of University of Washington and Arizona Sate University entrepreneurship programs. Purpose-driven business leader with global experiences and over a decade recruiting, managing, and leading talented multi-disciplinary teams to work on some of the world’s most pressing problems. Directly responsible for raising over $1M in biotech R&D funding annually and bringing technologies from the idea stage to the industrial or municipal scale. Deep financial modeling and investment experience on over $4 billion dollars of commercial and residential projects as well as due diligence deep dives into energy and industrial investments and acquisitions. Evan has previously won or been a finalist in the Mass Challenge, VentureWell E-Teams, Village Capital, Rice Business Plan Competition, MAC6 Conscious Capitalism, and CleanTech Open venture accelerator programs.

Benjamin Feuer, Department of Computer Science, Stanford

Ben Feuer is the post-training lead on the Marin project, a postdoctoral researcher in the lab of Percy Liang, a member of technical staff at OpenAthena, and a research fellow at Oumi. Before this, he co-lead the OpenThoughts-Agent project at Stanford with Ludwig Schmidt. Ben has completed an M.S. and Ph.D. at New York University supervised by Dr. Chinmay Hegde, and worked with AI startups Arthur AI and Abacus AI. He believes that AI is for everyone, and that it should be developed in the open.

Citas

Alfonzo, M., Guo, Z., Lin, L., & Day, K. (2014). Walking, obesity and urban design in Chinese neighborhoods. Preventive Medicine, 69(Suppl. 1), S79–S85. https://doi.org/10.1016/j.ypmed.2014.10.002

Alfonzo, M., & Leinberger, C. B. (2012). Walk this way: The economic promise of walkable places in metropolitan Washington, D.C. Brookings Institution. https://www.brookings.edu/articles/walk-this-waythe-economic-promise-of-walkable-places-in-metropolitan-washington-d-c/

Barbeau, S. J., Lozano, W., Turner, S., Sener, I. N., Maness, M., & Luong, T. (2022). Evaluating the accuracy of smartphone-based travel behavior data (Research Report No. 14). National Institute for Congestion Reduction. https://doi.org/10.5038/CUTR-NICR-Y1-3-1

Biswas, G., & Roy, T. K. (2023). Measuring objective walkability from pedestrian-level visual perception using machine learning and GSV in Khulna, Bangladesh. Geomatics and Environmental Engineering, 17(6), 5–34. https://doi.org/10.7494/geom.2023.17.6.5

Blondel, V. D., Decuyper, A., & Krings, G. (2015). A survey of results on mobile phone datasets analysis. EPJ Data Science, 4, 10. https://doi.org/10.1140/epjds/s13688-015-0046-0

Day, K., Boarnet, M., Alfonzo, M., & Forsyth, A. (2006). The Irvine–Minnesota Inventory to measure built environments: Development. American Journal of Preventive Medicine, 30(2), 144–152. https://doi.org/10.1016/j.amepre.2005.09.017

de Montjoye, Y.-A., Hidalgo, C. A., Verleysen, M., & Blondel, V. D. (2013). Unique in the crowd: The privacy bounds of human mobility. Scientific Reports, 3, 1376. https://doi.org/10.1038/srep01376

de Moraes Ferrari, G. L., Kovalskys, I., Fisberg, M., Gómez, G., & Rigotti, A. (2020). Socio-demographic patterns of public, private and active travel in Latin America: Cross-sectional findings from the ELANS study. Journal of Transport & Health, 16, 100788. https://doi.org/10.1016/j.jth.2019.100788

Diop, E. B., Chenal, J., Tekouabou, S. C. K., & Azmi, R. (2022). Crowdsourcing public engagement for urban planning in the Global South: Methods, challenges and suggestions for future research. Sustainability, 14(18), 11461. https://doi.org/10.3390/su141811461

Doiron, D., Setton, E. M., Brook, J. R., Kestens, Y., McCormack, G. R., Winters, M., Shooshtari, M., Azami, S., & Fuller, D. (2022). Predicting walking-to-work using street-level imagery and deep learning in seven Canadian cities. Scientific Reports, 12, 18380. https://doi.org/10.1038/s41598-022-22630-1

Du, R., Santi, P., Xiao, M., Vasilakos, A. V., & Fischione, C. (2019). The Sensable City: A survey on the deployment and management for smart city monitoring. IEEE Communications Surveys & Tutorials, 21(2), 1533–1560. https://doi.org/10.1109/COMST.2018.2881008

Ewing, R., & Cervero, R. (2010). Travel and the built environment: A meta-analysis. Journal of the American Planning Association, 76(3), 265–294. https://doi.org/10.1080/01944361003766766

Ewing, R., & Cervero, R. (2017). “Does compact development make people drive less?” The answer is yes. Journal of the American Planning Association, 83(1), 19–25. https://doi.org/10.1080/01944363.2016.1245112

Florida Department of Transportation. (n.d.). Traffic signal budgeting and costs – tables. https://wbt.dot.state.fl.us/ois/tsmo/TrafficSignalBudgetingCostsAccessibleTables.htm

Gebru, T., Krause, J., Wang, Y., Chen, D., Deng, J., Aiden, E. L., & Fei-Fei, L. (2017). Using deep learning and Google Street View to estimate the demographic makeup of neighborhoods across the United States. Proceedings of the National Academy of Sciences, 114(50), 13108–13113. https://doi.org/10.1073/pnas.1700035114

González, M. C., Hidalgo, C. A., & Barabási, A.-L. (2008). Understanding individual human mobility patterns. Nature, 453(7196), 779–782. https://doi.org/10.1038/nature06958

Haklay, M. (2010). How good is volunteered geographical information? A comparative study of OpenStreetMap and Ordnance Survey datasets. Environment and Planning B: Planning and Design, 37(4), 682–703. https://doi.org/10.1068/b35097

He, H., Guler, S. I., & Menendez, M. (2016). Adaptive control algorithm to provide bus priority with a pre-signal. Transportation Research Part C: Emerging Technologies, 64, 28–44. https://doi.org/10.1016/j.trc.2016.01.009

Hosseini, M., Miranda, F., Lin, J., & Silva, C. T. (2022). CitySurfaces: City-scale semantic segmentation of sidewalk materials. Sustainable Cities and Society, 79, 103630. https://doi.org/10.1016/j.scs.2021.103630

Ibrahim, M. R., Haworth, J., & Cheng, T. (2020). Understanding cities with machine eyes: A review of deep computer vision in urban analytics. Cities, 96, 102481. https://doi.org/10.1016/j.cities.2019.102481

Inglés-Romero, J. F., Doménech-Asensi, G., Cano-Baños, M.-D., & Morales-Esteras, V. (2018). Traffic metrics at urban intersections using a low-cost image processing approach. In Proceedings of the 12th International Conference on Distributed Smart Cameras (Article 9, pp. 1–7). Association for Computing Machinery. https://doi.org/10.1145/3243394.3243693

Ismagilova, E., Hughes, L., Rana, N. P., & Dwivedi, Y. K. (2022). Security, privacy and risks within smart cities: Literature review and development of a smart city interaction framework. Information Systems Frontiers, 24(2), 393–414. https://doi.org/10.1007/s10796-020-10044-1

Joy, J., McGoldrick, C., & Gerla, M. (2016). Mobile privacy-preserving crowdsourced data collection in the smart city [Preprint]. arXiv. https://doi.org/10.48550/arXiv.1607.02805

Le, H. T. K., Buehler, R., & Hankey, S. (2019). Have walking and bicycling increased in the US? A 13-year longitudinal analysis of traffic counts from 13 metropolitan areas. Transportation Research Part D: Transport and Environment, 69, 329–345. https://doi.org/10.1016/j.trd.2019.02.006

Liu, L., Wang, H., & Wu, C. (2016). A machine learning method for the large-scale evaluation of urban visual environment [Preprint]. arXiv. https://doi.org/10.48550/arXiv.1608.03396

Naik, N., Philipoom, J., Raskar, R., & Hidalgo, C. A. (2014). Streetscore—Predicting the perceived safety of one million streetscapes. In 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops (pp. 793–799). IEEE. https://doi.org/10.1109/CVPRW.2014.121

National Academies of Sciences, Engineering, and Medicine. (2019). Benchmarking and comparative measurement for effective performance management by transportation agencies. The National Academies Press. https://doi.org/10.17226/25365

Ning, H., Ye, X., Chen, Z., Liu, T., & Cao, T. (2022). Sidewalk extraction using aerial and street view images. Environment and Planning B: Urban Analytics and City Science, 49(1), 7–22. https://doi.org/10.1177/2399808321995817

Otero, B. G., & Verhulst, S. G. (2025, September 13). Why data governance and collaboration are essential for the future of urban digital twins. The GovLab Blog. https://blog.thegovlab.org/why-data-governance-and-collaboration-are-essential-for-the-future-of-urban-digital-twins

Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I. (2021). Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (Vol. 139, pp. 8748–8763). PMLR. https://proceedings.mlr.press/v139/radford21a.html

Sallis, J. F., Cerin, E., Conway, T. L., Adams, M. A., Frank, L. D., Pratt, M., Salvo, D., Schipperijn, J., Smith, G., Cain, K. L., Davey, R., Kerr, J., Lai, P.-C., Mitáš, J., Reis, R., Sarmiento, O. L., Schofield, G., Troelsen, J., Van Dyck, D., . . . Owen, N. (2016). Physical activity in relation to urban environments in 14 cities worldwide: A cross-sectional study. The Lancet, 387(10034), 2207–2217. https://doi.org/10.1016/S0140-6736(15)01284-2

Seiferling, I., Naik, N., Ratti, C., & Proulx, R. (2017). Green streets—Quantifying and mapping urban trees with street-level imagery and computer vision. Landscape and Urban Planning, 165, 93–101. https://doi.org/10.1016/j.landurbplan.2017.05.010

Toole, J. L., Colak, S., Sturt, B., Alexander, L. P., Evsukoff, A., & González, M. C. (2015). The path most traveled: Travel demand estimation using big data resources. Transportation Research Part C: Emerging Technologies, 58, 162–177. https://doi.org/10.1016/j.trc.2015.04.022

U.S. Department of Transportation, ITS Joint Program Office. (2012, January 17). The average installation cost per intersection of an adaptive traffic control system (ATCS) is $65,000. https://www.itskrs.its.dot.gov/2012-sc00249

Wang, C., Jia, R., Liu, X., & Song, D. (2024). Benchmarking zero-shot robustness of multimodal foundation models: A pilot study [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2403.10499

Wu, B., Yu, B., Shu, S., Liang, H., Zhao, Y., & Wu, J. (2021). Mapping fine-scale visual quality distribution inside urban streets using mobile LiDAR data. Building and Environment, 206, 108323. https://doi.org/10.1016/j.buildenv.2021.108323

Yan, Y., Wen, H., Zhong, S., Chen, W., Chen, H., Wen, Q., Zimmermann, R., & Liang, Y. (2024). UrbanCLIP: Learning text-enhanced urban region profiling with contrastive language-image pretraining from the web. In Proceedings of the ACM Web Conference 2024 (pp. 4006–4017). Association for Computing Machinery. https://doi.org/10.1145/3589334.3645378

Yüksekgönül, M., Bianchi, F., Kalluri, P., Jurafsky, D., & Zou, J. (2023). When and why vision-language models behave like bags-of-words, and what to do about it? In The Eleventh International Conference on Learning Representations (ICLR 2023). https://openreview.net/forum?id=KRLUvxh8uaX

Zhang, F., Salazar-Miranda, A., Duarte, F., Vale, L., Hack, G., Chen, M., Liu, Y., Batty, M., & Ratti, C. (2024). Urban visual intelligence: Studying cities with artificial intelligence and street-level imagery. Annals of the American Association of Geographers, 114(5), 876–897. https://doi.org/10.1080/24694452.2024.2313515

Vision Without Surveillance

Using Zero-Shot CLIP for Scalable Built Environment Feature Extraction

Autores/as

DOI:

Palabras clave: