bash - Remove enclosing quotes (both single and double) from a value in a keyvalue pair - Stack Overflow

I have a file file.txt as below, which contains key value pairs separated by a colon-space(: ).

key1: "Subject %D: %W warning(s), %V Violation(s)"
key2: 'sample: value'
key3: "value3"
key4: 'value4'

I want the output as below. Basically remove any single quote or double quote from value and replace colon-space with =.

key1=Subject %D: %W warning(s), %V Violation(s)
key2=sample: value
key3=value3
key4=value4

I have tried awk

awk -F': ' '{ gsub(/^'\''|'\''$/, "", $2); gsub(/^'\"'|'\"'$/, "", $2); print }'  OFS='='

but I am getting the output as below. It is failing when there is a colon-space in the value.

key1=Subject %D=%W warning(s), %V Violation(s)"
key2=sample=value'
key3=value3
key4=value4

Also tried the below sed which having the same problem as awk.

sed -i -e "s/: '/=/" -e "s/'$//" -e 's/: "/=/' -e 's/"$//' -e "s/: /=/" file.txt

key1=Subject %D=%W warning(s), %V Violation(s)
key2=sample=value
key3=value3
key4=value4

Is there is a way I can do this with few lines of code in unix?

I have a file file.txt as below, which contains key value pairs separated by a colon-space(: ).

key1: "Subject %D: %W warning(s), %V Violation(s)"
key2: 'sample: value'
key3: "value3"
key4: 'value4'

I want the output as below. Basically remove any single quote or double quote from value and replace colon-space with =.

key1=Subject %D: %W warning(s), %V Violation(s)
key2=sample: value
key3=value3
key4=value4

I have tried awk

awk -F': ' '{ gsub(/^'\''|'\''$/, "", $2); gsub(/^'\"'|'\"'$/, "", $2); print }'  OFS='='

but I am getting the output as below. It is failing when there is a colon-space in the value.

key1=Subject %D=%W warning(s), %V Violation(s)"
key2=sample=value'
key3=value3
key4=value4

Also tried the below sed which having the same problem as awk.

sed -i -e "s/: '/=/" -e "s/'$//" -e 's/: "/=/' -e 's/"$//' -e "s/: /=/" file.txt

key1=Subject %D=%W warning(s), %V Violation(s)
key2=sample=value
key3=value3
key4=value4

Is there is a way I can do this with few lines of code in unix?

• bash
• shell
• awk
• sed

Share Improve this question Follow edited Apr 1 at 19:12 John Kugelman 363k6969 gold badges553553 silver badges597597 bronze badges asked Jan 15 at 14:40 user29211006user29211006 5333 bronze badges 3

• 1 So you want to remove only the first :? – Arkadiusz Drabczyk Commented Jan 15 at 14:48
• 3 Can you have quotes inside the value? Inside or around the key? Can you have values not enclosed in quotes? Can you have values with only the opening quote? Only the closing quote? Different opening and closing quotes (simple vs. double)? If yes, what do you want to do in these cases? Please answer by editing your question, not in comments. – Renaud Pacalet Commented Jan 15 at 14:53
• Since you use -F': ' the colon after Subject %D marks the end of $2, so the rest of the line is not processed. – Barmar Commented Jan 15 at 22:59

Add a comment  | 

5 Answers 5

Sorted by: Reset to default 7

You may use this sed solution:

sed -E "s/: [\"'](.*)[\"']$/=\1/" file

key1=Subject %D: %W warning(s), %V Violation(s)
key2=sample: value
key3=value3
key4=value4

Regex pattern : [\"'](.*)[\"']$ matches colon then a space followed by " or '. Then we match and capture everything in group #1 until we match ending " or '. In the replacement we put = followed by the back-reference of group #1 back.

---

If you are looking for an awk solution then use:

awk -F ": [\"']|[\"']$" -v OFS='=' '{print $1, $2}' file

key1=Subject %D: %W warning(s), %V Violation(s)
key2=sample: value
key3=value3
key4=value4

This awk uses input field separator as colon then a space followed by " or ' OR else " or ' before the end. We use = as output field separator to put = between $1 and $2.

I would harness GNU AWK for this task following way, let file.txt content be

key1: "Subject %D: %W warning(s), %V Violation(s)"
key2: 'sample: value'
key3: "value3"
key4: 'value4'

then

awk 'BEGIN{FS=OFS=": "}{k=$1;gsub(/["\047]/,"");$1=k;sub(/: /,"=");print}' file.txt

gives output

key1=Subject %D: %W warning(s), %V Violation(s)
key2=sample: value
key3=value3
key4=value4

Explanation: I inform GNU AWK that : is both field separator and output field separator. I store 1st field value in variable k, remove quotes (note that you can't write ' directly as this would be treated as terminating command, thus I use GNU AWK way of writing and therefore \047), then I restore 1st field to initial value (this is important if you can have quotes in key value) and use sub to replace only 1st : in line, after that I print ready line.

(tested in GNU Awk 5.3.1)

As long as the key values have quoting styles consistent with the shown samples, then capturing groups are totally overkill for this task :

_1_='key1: "Subject %D: %W warning(s), %V Violation(s)"'
_2_="key2: 'sample: value'"
_3_='key3: "value3"'
_4_="key4: 'value4'"

printf '%s\n' "$_1_" "$_2_" "$_3_" "$_4_"  |

awk NF-- FS=': ["\47]|["\47]$' OFS==

key1=Subject %D: %W warning(s), %V Violation(s)
key2=sample: value
key3=value3
key4=value4

A Perl:

perl -lpE 's/^(\w+):\s*([\047"])([^\2]*)\2\s*$/$1=$3/' file

A Ruby:

ruby -lpe '$_=$_.sub(/^(\w+):\s*([\047"])([^\2]*)\2\s*$/,"\\1=\\3")' file

Either:

key1=Subject %D: %W warning(s), %V Violation(s)
key2=sample: value
key3=value3
key4=value4

It's easier to do it in two turns - one for single quoted lines, another for double quotes, but the logic is the same, given that input data is cat from file data.txt we could use Raku/Sparrow - code remains simple and easy to understand

# process single quotes lines
~regexp: ^^ \s* (\S+?) ":" \s+ \' (.*) \'


code: <<RAKU
!raku

for captures-full()<> -> $l {
  # patch original file in place
  replace(
    "data.txt",
    $l<index>,
    $l<data>[0] ~ "=" ~ $l<data>[1]
  );
}
RAKU

# process double quotes lines
~regexp: ^^ \s* (\S+?) ":" \s+ \" (.*) \"


code: <<RAKU
!raku

for captures-full()<> -> $l {
  # patch original file in place
  replace(
    "data.txt",
    $l<index>,
    $l<data>[0] ~ "=" ~ $l<data>[1]
  );
}
RAKU