Population migration derived from ACS 2011 5-year PUMS dataset

This is data release of working-age-population migration based on the ACS 2011 5-year PUMS. This article provides the same info as in my previous article Population migration derived from ACS 2011 5-year PUMS dataset.

The released spreadsheet table shows the population migration moved from each US State or foreign country into each US State, including in-state moves. It is to be emphasized that since these are based on sampling, the number is for references only. To get a sense of  possible errors, the MOE should be consulted. The spreadsheet can be accessed via Google Drive.

As an example, the spreadsheet show that, from 2007 to 2011, on average, there are about 46, 117, and 15 people per year with doctoral degree moved into Nebraska from France, China, and Jamaica respectively.

Educational Attainment of Nebraska's Working Age Population via 2016-12 ACS PUMS

20180504 Update: This is a data set that is prepared explicitly for Nebraska Coordinating Commission for Postsecondary Education (Nebraska CCPE) so that they can proceed with their 2018 Progress Report. The reason for this note is  to increase the chance so people can find the data because CCPE refused to cite this web article as the source of the data


This is a data release for Nebraska's working age population.  The working age is defined as 22 to 64 inclusive. The data is based on the PUMS (Public Use Micro Sample) data released by the US Census' American Community Survey.

The table below presented the number of people with various educational attainment with the age between 22 and 64.

Ed. Attainment	Population	Low(90%MOE)	Hi(90%MOE)	Percent
1. LssHsDgr	87,691	85,034	90,348	8.5%
2. HsDgrEqv	241,063	236,468	245,658	23.4%
3. SomeCllg	250,798	245,393	256,203	24.4%
4. AssctDgr	116,124	112,680	119,568	11.3%
5. BchlrDgr	234,181	228,870	239,492	22.7%
6. Mstr	71,874	69,336	74,412	7.0%
7. FP	16,806	15,641	17,971	1.6%
8. Drs	11,309	10,230	12,388	1.1%

Detailed data with sampling weights can be download from here located in Google Drive. The weights can allow users to aggregate the presented educational attainment levels.

Migration of Nebraska Working Age Population via 2016-12 ACS PUMS

20180504 Update: This is a data set that is prepared explicitly for Nebraska Coordinating Commission for Postsecondary Education (Nebraska CCPE) so that they can proceed with their 2018 Progress Report. The reason for this note is  to increase the chance so people can find the data because CCPE refused to cite this web article as the source of the data

This is a data release concerning Nebraska's working age population. The working age is defined as 22 to 64 years old, inclusive.

The table below shows the estimated net number of people that moved into Nebraska per year between 2012 and 2016 with age between 22 and 64.

Ed. Attainment	Net (In) Migration	Low(90%MOE)	Hi(90%MOE)
1. No HS Dgr	-203	-1056	650
2. HS Graduated	447	-680	1574
3. Some College	294	-1111	1699
4. Associate Dgr	366	-545	1277
5. Bachelor Dgr	-953	-2253	347
6. Graduate Dgr	-637	-1547	273

* 20180124 Number verified.
Detailed data with sampling weights can be download from Google drive at here.


.

Educational Attainment of Nebraska's Working Age Population via 2011-07 ACS PUMS

20180504 Update: This is a data set that is prepared explicitly for Nebraska Coordinating Commission for Postsecondary Education (Nebraska CCPE) so that they can proceed with their 2018 Progress Report. The reason for this note is to increase the chance so people can find the data because CCPE refused to cite this web article as the source of the data

For this data release, the working age is defined as 22 to 64, inclusive.

This is a quick release of Nebraska's data, the full data for every State will follow.

Last year I was summarizing data manually - basically, wrote database queries with a temporary mapping table that translates ACS' education attainment levels to our desired levels. This year, I formalized some features in the database and try to build queries based on those formalized features.

The Nebraska data is presented in the following table:

Ed. Attainment Level	Head Count	90% MOE	Possible Range
1. No High School Diploma	84,417	3,198	81,219 to 87,614
2. Has High School Diploma	258,046	4,404	253,641 to 262,450
3. Some College Exp. - No Degree	255,970	5,053	250,917 to 261,022
4. Associate's Degree	109,736	2,995	106,740 to 112,731
5. Bachelor's Degree	212,931	3,753	209,178 to 216,683
6. Master's Degree	59,012	2,146	56,865 to 61,158
7. First Professional Degree	17,250	1,020	16,230 to 18,269
8. Doctor's Degree	9,335	975	8,359 to 10,310

The released data can be accessed through Google drive here. The released data includes all estimates that can be used to combine different degree levels. For data related to other states, please follow this link.

Related articles: 
Nebraska Brain Drain Migration and Ed. Attainment, 2015 United States ACS 

Migration of Nebraska Working Age Population via 2011-07 ACS PUMS

20180504 Update: This is a data set that is prepared explicitly for Nebraska Coordinating Commission for Postsecondary Education (Nebraska CCPE) so that they can proceed with their 2018 Progress Report. The reason for this note is  to increase the chance so people can find the data because CCPE refused to cite this web article as the source of the data

This is simply a release of migration data for Nebraska working age population based on the ACS (American Community Survey) 2011-2007 5-year PUMS (Public Use Microdata Sample) data released by US Census Bureau.

Last year, after publishing the serials of articles about the population migration in the US, I re-examined what I did and spent times in revise the approach using more R codes than manually preparing and running SQL queries. This year, after comparing my R process for 2015-2011 and 2010-2006, I decide to restructure the R codes in an attempt to extract most of the common code to be shared and, hopefully, it will reduce the time spend in maintaining the code in the future. I intended to create R code to replace last year's process for education attainment too.

For this data release, the working age is defined as 22 to 64, inclusive.

Education Degree	Net (In) Migration	90% MOE	Possible Range
1. No High School Diploma	2,130	1,037.3	1092 to 3167
2. Has High School Diploma	35	1,247.1	-1213 to 1282
3. Some College Exp. - No Degree	1,501	1,493.2	7 to 2994
4. Associate's Degree	153	824.8	-672 to 977
5. Bachelor's Degree	89	1,260.9	-1172 to 1349
6. Graduate Degree	-1,733	935.0	-2669 to -798

As can be seen from the above table, for every year during that five-years period, there are, in net, estimated 1,733 people with graduate degree moved out of Nebraska. Since the number is derived from sampling, with 90% of certainty, the true number can lie between 2,669 and 798. So it is very likely (90% certainty) that Nebraska loses about 2,669 to 798 people with graduate degree every year during the five year period.

For population with bachelor degree, the net migration pattern isn't as clear cut as those with graduate degree since, with 90% certainty, the net can vary from 1,172 moving out to 1,349 moving in.


The released data file can be accessed here through Google Drive. The released data includes all weights that is needed to combine education categories if so desired. For data concerning other states, please follow this link.


Related articles: 
Nebraska Brain Drain Migration and Ed. Attainment, 2015 United States ACS 

Adult College Enrollment - IPEDS data

As described in my previous articles, I have been working on importing all IPEDS (Integrated Postsecondary Education Data System) data into my database. After spending years on this project, I was able to import about 15 years' worth of IPEDS into the database with verification. There are still few files that, without major efforts, would be hard to handle properly - these are basically long lines with embedded new line characters in the line.

That being said, I was eager to run a test case with these IPEDS data. As the fate has presented itself, my previous articles were about adult college enrollment, and it happens that the college enrollment can be approximated from the IPEDS enrollment data.

For this article, the IPEDS fall (semester) enrollment data were first examined via my R interface code, which allows searching and checking definitions across data years. In this particular case, the R code reveals that, at 2009, the 'first professional' enrollment level disappeared from the level definition. By examining the IPEDS documentation, it is verified that from 2009 and on, the  first professional enrollment is to be reported in the graduate enrollment level. Since I am a kind of familiar with the IPEDS data collection, I knew the enrollment age data were not mandatory for even-number years. If not, a quick R code that checking the total for each year should have revealed that.

Since in IPEDS, data were only tagged with college id (unitid), extra steps were needed to tag the data with attributes from the college. These attributes are made available through the so called 'Institutional Chararcteristic' survey. Whit this survey, colleges can be tagged with control (Public/Private), level (4 or 2 year college), location (state/address...). For this project, it was found that, in 2011, there were 3 institutions did not reported appropriate information for the 'institutional characteristic' survey. Luckily, two of them were available from other years. To preserve most data, we fixed the two with info from other year and coded the third one with special code so that we can include them if we so desire. For this article, we include all institutions that were collected by IPEDS and this include institutions that located on US territories and miscellaneous islands. To list a few, this includes AS (America Samoa), GU (Guam), PR (Puerto Rico), MH (Marshall Island) ... etc.

With previous adult college enrollment article in mind, under-graduate enrollment from the IPEDS was considered a better approximation to those from the ACS data.


Examining the IPEDS age data, it is noticed that not all data were collected with equal age span. For example, data are collected with age categories like 18 to 19, 22 to 24, 25 to 29 ... etc. Presenting age data directly with with these age categories results in the following chart and the chart can trick reader to think that there is a bump in the age distribution which sure not look like the age distribution presented in my previous article.

Age distribution using IPEDS age categories

A better approach to resolve this would be using the average head count for each age category instead. Better yet, you can assign the average to each age in the category to provide a better representation in terms of age axis.

In this article, an average assigned to the category is used. To approaching the college enrollment data in my previous ACS based article, we presented the age distribution with the total enrollment, the sum of both full-time and part-time students. As shown in the following graph, it can be seen that the curve exhibits a familiar monotonic decreases after the primary peak around college graduation.

Age distribution for total fall enrollment using average for each age category

Since the IPEDS data also allow the separation of data with full-time, and part-time, it is worth the efforts to examine these characters too. The overall (sum over states) full-time distribution can be seen in the chart below.

Age distribution for full-time fall college enrollment using average for each age category

A typical age distribution for a state (NE) can be seen below. For most state, the only difference is whether the age group 18 to 19 or the age group 20 to 22 is the highest. The full-time fall enrollment age distribution for Utah, however, show a very different distribution - see chart below. This may related to the Mormon missionary program but more evidence from other survey or data elements may be needed.

A typical state (NE) age distribution for full-time enrollment using average for each category

Full-Time fall enrollment age distribution for the state of Utah

The IPEDS universal total for part-time fall enrollment can be seen in the chart below. Comparing to full-time and total enrollment, it clearly show a distinctive age distribution. For some states, their part-time fall enrollment are similar to that of the IPEDS universal total as shown below (NE). There are, however, another set of states (e.g. CA, FL, GA, ... etc.) that shows a quite different age distribution pattern. Part-time students in these states seem to take a break from school (to work?) and come back to enroll in school later.

IPEDS universal age distribution for part-time fall enrollment

Part-Time fall enrollment for the State of Nebraska

Part-Time fall enrollment for the State of California

Examining the IPEDS universe part-time enrollment, trending by years, we noticed that there were more younger kids in recent years. By presenting these same data in percentages, it shows that, proportionally, elder adult were taking smaller share of the part-time enrollment in recent years.

Age distribution of part-time students in percents

Updated 20180525: A Kickstarter project has been created that will allow average data user to obtain these kind of IPEDS data.

Education Enrollment and Adult College Enrollment by Age - 2016 ACS

In my previous article, we look at the total adult college enrollment for each state. In this, I would like to take one step further to look into the enrollment by age. In this article, ages are presented in an interval of 3. An age label of 25 is actually the sum of age 24, 25, and 26.

Again, the analysis is post on my public tableau website.

The first chart presents few basic numbers that provided background information for discussion. In the chart, the variable zTtl represents the total population. When select all states, the Millennial's bump at age 25 can clearly be seen. The curve zAttnmntUndr+ represent all people with education attainment of undergraduate degree (associate or bachelor) or above. As shown in the chart, the curve first take a large gain at age 25 when most college students obtained their undergraduate degrees. The curve then peaked at age 31 when few more students finishing their undergraduate degrees and others obtained their graduate degrees. However, keep in mind that, in this age chart, the total population trend also contributes.

Education Attainment and Enrollment by Age - United States ACS 2016

The line zAttnmntNoCllgDgr represents the number of people do not obtained undergraduate degree or above. As can be seen in the chart, the line begins to drop once students begin to obtain their undergraduate degrees. Again, keep in mind, the population trend has its effect as show in the bump around age 55, where, due to the population bump,  the population do not have college degree also show a bump. The remaining two green-blueish lines display the college undergraduate (zEnrllUndr) and graduate (zEnrllGrdt) enrollment. These two lines peaked at age 19 and 25 respectively.


The second chart provide basic information on college-education attainment by age for each state. The variable zpctAttnmntUndr+ shows the percent of population obtained undergraduate degree and above. With all states selected, the United States shows a highest rate of 46% at the age of 34. The other two lines on this chart, zpctAttnmntUndr and zpctAttnmntGrdt, represent the percent of population with the highest degree attainment of undergraduate or graduate degrees.

Education Attainment By Age - United States ACS 2016

The third chart displays two education attainment lines from the first chart (zAttnmntUndr+, and zAttnmntNoCllgDgr) along with a college enrollment line (zNoUndrEnrllUndr+). The college enrollment line counts all students enrolled in both undergraduate and graduate but haven't obtain an undergraduate degree yet. By taking the ratio of zNoUndrEnrllUndr+ against zAttnmntNoCllgDgr we arrived at the forth chart.

Education Attainment and Enrollment By Age - United States ACS 2016

As mentioned above, the forth chart display the ratio between zNoUndrEnrllUndr+ and zAttnmntNoCllgDgr. It shows the fraction of non-college degree earner enrolled in college classes. Conferred to chart three, since at the age of 18, 19, and 20, there were relatively few people (313,517) obtained college degrees, the peak at the age 19 shows that, in US, about 50% of that age group went to college. Also observed on the chart is that, after the peak, the percentage of non-college-degree bearer enrolled in college decreased monotonically. Meaning that the older the people are, the less likely they will enrolled as an adult student. For Adult college enrollment at age 25 (aggregates age 24,25, and 26), the rate are about 15% nationally, and dropped fast to 9% at age 28 ( aggregates 27, 28, and 29).

Non-Degree Bearer Enrollment By Age - United States ACS 2016

A Look At the Adult College Enrollment by State - 2016 ACS

After importing the ACS (American Community Survey) 2016 1-Year PUMS (Public Use Micro Sample) data, the question is what come next? Just holding to sets of data doesn't do anyone any good except taking up disk spaces - in this case, it takes about 2.9GB including both the personal and housing unit data.

With the push of Obama's call for increasing college attainment of 60% by 2020, the adult college enrollment can be an important factor. * Obama, however, did not limited his view to associated degree and above. Researchers have been trying to include all sorts of certificates on their way to compare to the 60% goal.

In this article, we try to measure what percentage of the adult who had not obtained an undergraduate degree is enrolled in college. We defined the adult as anyone who is 25 years old or older. This approach, basically give the traditional high school graduates 6 years to obtain their undergraduate degree before been considered in our measure.

As would be presented in a separate article, our definition of 'adult college enrollment' may not match that mental picture of some of our readers since some reader may only consider individuals that rejoin the college after years of absent as the 'adult college enrollment'. While with our definition, traditional students taking longer than 6 years to obtain their first college are included.

Our result of processing the 2016 ACS 1-Year PUMS is presented at Tableau's public use web site.

In the first chart, we presented three numbers that were used for each state. The first number, NoUndrDgr, is the number of adults 25 years old or older that did not obtained an undergraduate degree (associates or bachelor) yet.The second number, WorkOnUndrDgr, for a state is the number of adults that were defined in the first number that are enrolled in undergraduate college programs. The third number, WorkOnGrdtDgr, for a state is the number of adult that were defined in the first number that are enrolled in college graduate program. Even though traditional wisdom doesn't think the third number is possible, for some states these numbers are perceivable. One possible case is due to medical schools. Because some medical school only require appropriate undergraduate course work and standard testing results to be qualified as applicants, it become possible for these students to enrolled in graduate medical program before obtaining bachelor degree. The other possibilities become possible since some colleges begin to thread bachelor degree and the graduate degree together so that students are enrolled for graduate degree while work toward finishing up bachelor degree with courses that credited for both degree. There were cases, the students will be awarded both degree at the end of the program and resulting graduate school enrollment before finishing up bachelor degree.

The Number for Adult College Enrollment

In the second chart we present the percent of students who were 25 years old or older that had no received an undergraduate degree that are enrolled in college undergraduate program. The chart is intended to rank the states based on the percent of adult college enrollment for adult of 25 years old or older that had not obtained their first undergraduate degree. In this chart, we ignored the third number presented in the first chart just to keep the definition clearer and consider those third number exception cases. The second chart also display the 90% MOE, which represent the range of possible sampling errors.

Adult College Enrollment Rate with 90% MOE

The third chart presents the Adult College Enrollment Rate in the map format.

Adult College Enrollment Rate by States

The forth chart provide the number in a table format and allows user to easily compare selected states. 

Table for Comparing Selected States

The following table lists all states in ranking order.

ST	% Enrolled	90% MOE
UT	3.9%	0.4%
HI	3.8%	0.6%
CA	3.7%	0.1%
DC	3.6%	1.0%
NM	3.4%	0.5%
AK	3.4%	0.8%
RI	3.3%	0.5%
WA	3.2%	0.2%
MD	3.1%	0.3%
CO	3.1%	0.3%
VA	3.0%	0.2%
AZ	3.0%	0.2%
OR	3.0%	0.3%
GA	3.0%	0.2%
NC	2.9%	0.2%
TX	2.9%	0.1%
NV	2.8%	0.3%
ID	2.7%	0.5%
DE	2.7%	0.6%
CT	2.7%	0.3%
KS	2.7%	0.3%
MN	2.6%	0.3%
MI	2.6%	0.2%
SC	2.5%	0.2%
WY	2.5%	0.7%
MA	2.5%	0.2%
FL	2.5%	0.1%
WI	2.4%	0.2%
NH	2.4%	0.4%
IL	2.4%	0.2%
LA	2.4%	0.2%
OH	2.3%	0.2%
NY	2.3%	0.1%
IN	2.3%	0.2%
OK	2.2%	0.3%
MO	2.2%	0.2%
AL	2.1%	0.2%
MT	2.1%	0.5%
NJ	2.1%	0.2%
NE	2.1%	0.4%
TN	2.1%	0.2%
MS	2.0%	0.3%
AR	2.0%	0.3%
ME	2.0%	0.4%
KY	1.9%	0.2%
ND	1.9%	0.8%
VT	1.7%	0.6%
SD	1.7%	0.5%
IA	1.7%	0.2%
PA	1.6%	0.1%
WV	1.5%	0.3%

Continued struggle with IPEDS data

As mentioned earlier, works in automate data importing take enormous amount of dedication and efforts. For those do not appreciate, there is really no need to share the knowledge with them.

Here is an example that demonstrate the kind of work and dedication is needed to solve just one problem that I run into while importing IPEDS data.

One problem format I run into in some IPEDS csv file is: 
...,""some text quoted with two double quote"", ...

As a human, we know this line break the csv convention and most likely any csv file importing program is going to fail.

As a data user, I got few resolutions to consider. If I am only dealing with this file, the fastest way is to just open the file in text editor and modify the line so that the csv file can be imported into my application. If you are thinking this way, most likely you are a data analyst and probably think this is how things should be handled. Since you are higher up in the data food chain, likely, have not appreciate the work and thought of IT professions.

IT professions are likely to view the situation from a much broad point of view and ask questions like: What if this is an error exist in ACS' csv file? - If you know the size of a general ACS' csv file, you will realize that there probably very few text editor can effectively open the file, let alone to locate the error line and fix it.

IT professions may also ask: What if there are other csv files also have this problem? How can I handle this automatically?

One tool a lot of IT profession know about is the sed program. To use the sed to fix this problem it is straight forward:
  sed 's/,"("[^"]{2,}")"/,\1/g; ' InCsv

Unfortunately, if you want to invoke this with VBA, the command become much more complicate:
  Cmd.exe /c ^"sed ^-r ^-n ^'^{s^/^,^"^(^"^[^^^^^"^]^{2^,^}^"^)^"^/^,^\1^/g^; p^;^}^' InCsv ^"

Let's just say this, if you have no clue what we are talking about here, you should appreciate the work of IT professions.

Brain Drain - 2015 ACS State Migration for Working Age

Updated on Feb. 27, 2017:
An attempt has been made to rank states by brain drain indexes based on migration. See article Brain drain - ranking and analysis with 2015 ACS data.

As will be discussed in my upcoming article about various possible conception/perceptions about the idea of 'brain' in the context of 'brain drain', one of the definition or interpretation would be the education attainment of the workforce, hence, the age range of 22 to 64 years old.

As mentioned in my previous article: Population migration derived from ACS 2015 5-year PUMS dataset, I was in the process of producing a more detailed result from the ACS 2015 5-year PUMS dataset and here it is. For this article, we ignored the in-migration from foreign counties, which was included in the previous article. Time allowed, we will look into foreign country migration in details.

Basically, we look at all samples with age between 22 and 64 years old in the PUMS file along with each sample's education attainment level and the state of residing a year ago. By analyzing these data, we can estimate the number of people moving in and out of a state and with what kind of education attainment level.

It happened that I was attending a Tableau promotion meeting recently and decided to give it a try even though I would have preferred an open source solution, which I did try to look up, but did not have enough time to evaluate them yet.

The rest of this article will simply provide notes to the presentation since I have the baggage of an old IT worker that abbreviates almost everything.

First, the cite of the data source: ACS 2015-2011 5-year PUMS file processed by Dr. Duncan Hsu.

The Brain Drain Migration between states presentation can be found at Tableau Public and below are some of the summaries: 

The first tab/page/slide, "Map - Migrated to To_State", is the in-migration map for the state of interest specified by the right-hand side dropdown control: the To_State. The map will show the number of people moving from each state to the To_State, with the education attainment level you specified by the second dropdown list labeled EdAttnmnt. To see the numbers, hover your mouse above a state of interest. For example, the following chart show that there were 724 people moved from Kansas to Nebraska, which was selected as the 'To_State'. Possible values for the EdAttnmnet dropdown are: Less than High School Degree (LssHsDgr), High School Degree or Equivalent (HsDgrEqv), Some College Experience/Course-work but no degree (SomeCllg), Associate Degree (AssctDgr), Bachelor Degree (BchlrDgr), Master Degree (Mstr), First Professional Degree (FP), and Doctor's Degree (Drs).

In-Migration to Nebraska

The second tab, "Map - Migrated out From_State", is the out-migration map for the state of interest. Operational wise, this is very similar to the first tab. In the map below, it shows that there were 421 people migrated to Iowa from Nebraska, which was selected as the 'From_State'.

Out-Migration from Nebraska

The third tab, "Map - Net migration", shows the net migration. By hover over each state, it shows three numbers, the HdCnt (head count; negative for out-migration and positive for in-migration) and the upper and lower bound for the 90% confidence level. The dropdown to the right displays 6 education attainment levels with the Graduates (Grdts) encompassing  Master, First Professional, and Doctor's degree.

Net-Migration for Nebraska

The fourth tab, is the net migration bar chart for each state where the bar indicated the Margin of Error (MOE) at the 90% confidence level. Again, dropdowns are on the  right.

Net-Migration with 90% MOE

The fifth tab provides the data used in the fourth tab in table format with the upper and lower bound of the 90% MOE.

Net-Migration for Nebraska's neighboring states

The Sixth tab allows selecting states with the map.

Selecting States with Map

Population migration derived from ACS 2015 5-year PUMS dataset

Now that we have got all the data imported, let's have some fun.

For those of you who knows me, I have been an advocate for open source movement for a while now. The statistic software I preferred to use have been the R. However, I had not spent a lot of my time on R - I think we all understand that people got a lot of things to do and we revisit a tool when we needed to.

Couple months ago, I spent my spare time and wrote quite a bit of code in R and I thought that I will be right at home when I decided to take on this migration project. Boy, am I wrong about this... gosh. Well, spent almost whole day and end up fixing some of the bugs - well, not really a bug but because I have decided to include the NA definition into my definition database, it caused some problem when referencing these definitions from my old code. Anyway, got it fixed but did not really use the R.

Well - my IT training kicked in - I realized that instead of using the statistic software for this project, a few SQL statement will largely simplify the task to nothing. Come to think about this, the SQL not only easier, it actually run much faster - Database is designed to run from hard disk, it is not like most statistic software will load all the data into memory and tied up the computer resources. By the way, a while back I have this idea of using database as my statistic software. I actually check out MS SQL documentation on customer functions and, do you know what, it is totally possible. Now, the question is who is going to take on this project.

Anyway, I end up running few SQL statements and dumping it into Excel with a bunch of formula - sorry, I haven't really invested in the Open Office yet.

OK. Let's get back to the topic. American Community Survey is conducted by US Census Bureau in an annually basis. The PUMS file is sampled from the collected data and allows user to use these sample to derive results that weren't readily tabulated by the US Census Bureau.

Inside the ACS survey, there is a question that asked respondents where they lived a year ago. Based on this question, we can look into the PUMS data and derive some useful information from it. One of the interesting application of this question is when it is combined with the education attainment info of the respondents. This allowed data analysts to see that, for people moving out of a state, what kind of education these people acquired and, hence, the brain drain if highly educated people left a state.

Click here for the resulting file - please noted that for any result derived from sampling, there are associated errors - this file does not come with the 'margin of errors', which describes the range the real value may lie. In our case, with large enough margin of errors, the real value for an in-migration could end up in negative and, hence, associated with the idea of  an out-migration. So, the file is for references only. The author is working on consolidate some of the categories and, hopefully, can report some data with reasonable 'margin of error'.




 

ACS 2015 5-year PUMS for database/IT professional

Continue with the ACS PUMS database project, the task is to import the 5-year ACS PUMS product of 2015.

Comparing Census's 2015 Data Dictionary file (PUMS_Data_Dictionary_2011-2015.txt) to that of 2014, we, again, noticed that Census' added leading spaces to a lot of lines to, possibly, make the file more readable for 'human' users. Following the step of processing the 2015 1-year file, I removed those leading spaces via sed before processing it with my definition processor.

Processing the Data Dictionary file with my program, it yields the following parsing errors. Some of them are clearly unintended errors, others may just because Census did not spend time and efforts to establish clear syntax rules so that their products can be machine friendly. Here are the parsing errors:


      ADJINC
        value 1001264, the blank after '1001264' is actually an A0h instead of
          20h.
      TEN
        just before TEN, a two line 'NOTE:'
      PERSON RECORD
        - no blank line after the 'PERSON RECORD' section mark
      ADJINC
        value 100264, the blank after '1001264' is actually an A0h instead of
          20h.
      GCL
        just before GCL, a two line 'NOTE:'
      FPINCP
        no empty line before FPINCP

The A0h one is really interesting. For those of interest, A0h is a NBSP character used in HTML. Without a good hex editor, it takes me a lot of efforts to figure out what is going wrong.